Answers.Body: <p><em>This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment.</em></p>
  
  <hr>
  
  <p>Yes you will very likely need to write software to represent your environment in some standard way as a Reinforcement Learning (RL) environment. Depending on details, this may be trivially easy or it might be quite involved.</p>
  
  <p>An environment in RL must have the following traits in general, in order to interface with RL agent software:</p>
  
  <ul>
  <li><p>A state representation. This will typically be an object or array of data that matches sensor readings from the real environment. It is important to RL that the state has the <a href="https://en.wikipedia.org/wiki/Markov_property" rel="nofollow noreferrer">Markov property</a> so that predictions of value can be accurate. For some environments that will mean calculating derived values from observations, or representing a combined history of last few observations from sensors as the state.</p>
  
  <ul>
  <li><p>The state can either be held inside an internal representation of the environment, which is a typical object-oriented approach, or it can be passed around as a parameter to other functions.</p></li>
  <li><p>A simple state might just be a fixed size array of numbers representing important traits of the environment, scaled between -1 and 1 for convenience when using it with neural networks.</p></li>
  </ul></li>
  <li><p>An action representation.</p>
  
  <ul>
  <li>A simple action representation could just be an integer which identifies which of N actions has been chosen, starting from 0. This allows for a basic index lookup when checking value function estimates.</li>
  </ul></li>
  <li><p>A reward function. This is part of a problem definition, and you may want to have that code as part of the environment or part of the agent or somewhere in-between depending on how likely it is to change - e.g. if you want to run multiple experiments that optimise different aspects of control but in the same environment, you may make a totally separate reward calculation module that you combine at a high level with the agent and environment code.</p>
  
  <ul>
  <li><a href="https://ai.stackexchange.com/a/12299/1847">Reward function design is a complex subject</a>. However, the basics are that this should always be a single real valued number (i.e. a floating point number in most programming languages).</li>
  </ul></li>
  <li><p>A time step function. This should take an action choice, and should update the state for a time step - returning the next state, and the immediate reward. If the environment is real, then the code will make actual changes (e.g. move robot arm), potentially wait for the time step to elapse, then read sensors to get the next state and calculate reward. If the environment is simulated, then the code should call some internal model to calculate the next state. This function should call the proprietary software you have been provided for your task.</p></li>
  </ul>
  
  <p>If actions available depend on the current state, then code for that could live in the environment simulation or the agent, or be some helper function that the agent can call, so it can filter the actions before choosing one.</p>
  
  <p>If you are working in Python, to help make this more concrete, and follow an existing design, see <a href="https://stackoverflow.com/questions/45068568/how-to-create-a-new-gym-environment-in-openai">"How to create a new gym environment in OpenAI?"</a> on Stack Overflow. The Open AI environments all follow the same conventions for environment definitions, which helps when writing agents to solve them. I also recommend finding an Open AI Gym environment that seems similar to your problem, seeing how that works and trying to train an agent to solve it.</p>
  
  <p>There may still be work to match your environment to an agent that can solve it. That depends on what agent software you are using.</p>
  
  <p>Even if you write your own agent software, it helps to separate out the environment code like this. The environment is the problem to solve, and the RL agent is one way to search for a solution to it. Keeping those parts separate is a useful design that will allow you to try different types of agent and compare their performance for example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-29T13:20:23.087Z
Answers.Text: This answer at least provided me with concrete direction to look. I looked into that StackOverflow answer on creating a custom environment. I had a doubt if using OpenAI gym is the correct approach. Now that I know it correct approach, I can invest my time to learn how to create custom environments.
Answers.UserDisplayName: NewToCoding
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-29T13:53:59.090Z
Answers.Text: @VamshiPulluri: I would not say that OpenAI gym is **the** correct approach, but provided you can work in Python, it is very likely to be **a** useful approach for you. The best thing is that there are many examples of agents that solve Open AI gym environments, and you will be able to use those more directly as resources for the rest of your project
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-29T12:42:18.670Z
Answers.LastActivityDate: 2019-05-29T12:57:00.323Z
Answers.LastEditDate: 2019-05-29T12:57:00.323Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am a newbie in reinforcement learning working on a college project. The project is related to optimizing the hardware power. I am running proprietary software in Linux distribution (16.04). The goal is to use reinforcement learning and optimize the power of the System (keeping the performance degradation of the software as minimum as possible).</p>
  
  <p>For this, I need to create a custom environment for my reinforcement learning. From reading different materials, I could understand that I need to make my software as a custom environment from where I can retrieve the <code>state</code> features. <code>Action</code> space may include instructions to Linux to change the power (I can use some predefined set of power options).</p>
  
  <p>The proprietary software is a cellular network and the <code>state</code> variables include <code>latency</code> or <code>throughput</code>. To control the power <code>action space</code>, <a href="https://github.com/kentcz/rapl-tools" rel="nofollow noreferrer">rapl-tools</a> can be used to control CPU power.</p>
  
  <p>I just started working on this project and everything seems blurry. What is the best way to make this work? Is there some tutorials or materials that would help me make things clear. Is my understanding of creating a custom environment for reinforcement learning true?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-29T11:48:51.423Z
Comments.Text: Could you please clarify whether your college project will work with a *simulation* of the cellular network, or a real instance using real power and providing real services? Either way, do you already have access to an API or the simulation or for controlling the real environment, or are you expected to write that too as part of your project?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-29T12:22:44.623Z
Comments.Text: @Neil Slater The action space include controlling the CPU power with Intel's RAPL interface. I can use rapl-tools to control CPU power.
Comments.UserDisplayName: NewToCoding
Tags: <reinforcement-learning><environment>
Title: How to create a custom environment for reinforcement learning
UserDisplayName: NewToCoding
ThreadText: # How to create a custom environment for reinforcement learning
  
  (Asked by: username_0 on 2019-05-29)
  
  I am a newbie in reinforcement learning working on a college project. The project is related to optimizing the hardware power. I am running proprietary software in Linux distribution (16.04). The goal is to use reinforcement learning and optimize the power of the System (keeping the performance degradation of the software as minimum as possible).
  
  For this, I need to create a custom environment for my reinforcement learning. From reading different materials, I could understand that I need to make my software as a custom environment from where I can retrieve the `state` features. `Action` space may include instructions to Linux to change the power (I can use some predefined set of power options).
  
  The proprietary software is a cellular network and the `state` variables include `latency` or `throughput`. To control the power `action space`, [rapl-tools](https://github.com/kentcz/rapl-tools) can be used to control CPU power.
  
  I just started working on this project and everything seems blurry. What is the best way to make this work? Is there some tutorials or materials that would help me make things clear. Is my understanding of creating a custom environment for reinforcement learning true?
  
  **Tags:** reinforcement-learning, environment
  
  > **Comments:**
  > **username_1**: Could you please clarify whether your college project will work with a *simulation* of the cellular network, or a real instance using real power and providing real services? Either way, do you already have access to an API or the simulation or for controlling the real environment, or are you expected to write that too as part of your project?  
  > **username_0**: @username_1 The action space include controlling the CPU power with Intel's RAPL interface. I can use rapl-tools to control CPU power.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-29. Score: 4
  
  *This answer assumes that your "proprietary software" is a simulation of, or controller for a real environment.*
  
  ---
  
  Yes you will very likely need to write software to represent your environment in some standard way as a Reinforcement Learning (RL) environment. Depending on details, this may be trivially easy or it might be quite involved.
  
  An environment in RL must have the following traits in general, in order to interface with RL agent software:
  
  * A state representation. This will typically be an object or array of data that matches sensor readings from the real environment. It is important to RL that the state has the [Markov property](https://en.wikipedia.org/wiki/Markov_property) so that predictions of value can be accurate. For some environments that will mean calculating derived values from observations, or representing a combined history of last few observations from sensors as the state.
  
    + The state can either be held inside an internal representation of the environment, which is a typical object-oriented approach, or it can be passed around as a parameter to other functions.
    + A simple state might just be a fixed size array of numbers representing important traits of the environment, scaled between -1 and 1 for convenience when using it with neural networks.
  * An action representation.
  
    + A simple action representation could just be an integer which identifies which of N actions has been chosen, starting from 0. This allows for a basic index lookup when checking value function estimates.
  * A reward function. This is part of a problem definition, and you may want to have that code as part of the environment or part of the agent or somewhere in-between depending on how likely it is to change - e.g. if you want to run multiple experiments that optimise different aspects of control but in the same environment, you may make a totally separate reward calculation module that you combine at a high level with the agent and environment code.
  
    + [Reward function design is a complex subject](https://ai.stackexchange.com/a/12299/1847). However, the basics are that this should always be a single real valued number (i.e. a floating point number in most programming languages).
  * A time step function. This should take an action choice, and should update the state for a time step - returning the next state, and the immediate reward. If the environment is real, then the code will make actual changes (e.g. move robot arm), potentially wait for the time step to elapse, then read sensors to get the next state and calculate reward. If the environment is simulated, then the code should call some internal model to calculate the next state. This function should call the proprietary software you have been provided for your task.
  
  If actions available depend on the current state, then code for that could live in the environment simulation or the agent, or be some helper function that the agent can call, so it can filter the actions before choosing one.
  
  If you are working in Python, to help make this more concrete, and follow an existing design, see ["How to create a new gym environment in OpenAI?"](https://stackoverflow.com/questions/45068568/how-to-create-a-new-gym-environment-in-openai) on Stack Overflow. The Open AI environments all follow the same conventions for environment definitions, which helps when writing agents to solve them. I also recommend finding an Open AI Gym environment that seems similar to your problem, seeing how that works and trying to train an agent to solve it.
  
  There may still be work to match your environment to an agent that can solve it. That depends on what agent software you are using.
  
  Even if you write your own agent software, it helps to separate out the environment code like this. The environment is the problem to solve, and the RL agent is one way to search for a solution to it. Keeping those parts separate is a useful design that will allow you to try different types of agent and compare their performance for example.
  
  > **Comments:**
  > **username_0**: This answer at least provided me with concrete direction to look. I looked into that StackOverflow answer on creating a custom environment. I had a doubt if using OpenAI gym is the correct approach. Now that I know it correct approach, I can invest my time to learn how to create custom environments.  
  > **username_1**: @VamshiPulluri: I would not say that OpenAI gym is **the** correct approach, but provided you can work in Python, it is very likely to be **a** useful approach for you. The best thing is that there are many examples of agents that solve Open AI gym environments, and you will be able to use those more directly as resources for the rest of your project  
  
  ---
  
  