Answers.Body: <blockquote>
  <p>I am wondering <strong>how to generate datasets when the environment is not as simple as a tic-tac-toe</strong> or a maze problem</p>
  </blockquote>
  <p>There is no difference in concept, which is why tic-tac-toe and maze problems are used to teach.</p>
  <p>As you have noted, the main difference between reinforcement learning (RL) and supervised learning is that RL does not use labeled datasets. If you are using SARSA then you would not expect to use any record of previous experience either because SARSA is designed to work on-policy and online - which means that data needs to be generated <em>during</em> training. Training data for SARSA is typically stored only temporarily before being used, or is used immediately (you might keep a log of it for analysis or to help document your thesis, but that log will not be used for further training by the agent). This is different to Q-learning and DQN, which could in theory make use of longer-term stored experience.</p>
  <p>You have two main choices for acquiring data:</p>
  <ul>
  <li><p>Use a real environment. In your case, set up 15 mobile users and 3 edge servers. Instrument the environment to collect state and reward data for the agent. Implement the agent as the real decision maker in this environment.</p>
  </li>
  <li><p>Simulate the environment. Write a simulation that models user behaviour and server loading. Instrument that to provide state and reward data, and integrate your learning agent with it. Typically the agent will call the environment's <code>step</code> function, passing the action choice as an argument and receiving reward and state data back.</p>
  </li>
  </ul>
  <p>If you can simulate the environment, this is likely to be preferable to you since you will likely use less compute resources (than 3 servers and 15 mobile phones) and can run the training faster than real time. Deep reinforcement learning can use a large amount of experience to converge on near-optimal policies, and fast simulations can help because they generate experience faster than reality.</p>
  <p>You can also do both approaches. Train an initial agent in simulation, then implement the real version once it reaches a good level of performance in simulation. You can even have the agent continue to learn and refine behaviour in production. Given that you are working with SARSA, this may be an important part of the intent of your project, that the agent continues to adapt to changes in user behaviour and server load over time. In fact this is a key advantage of SARSA over Q-learning, that it should be more reliable and safe to use in such a continuous learning scenario deployed to production.</p>
  <blockquote>
  <p>and <strong>what the experience means</strong>.</p>
  </blockquote>
  <p>The experience in reinforcement learning is the record of states, actions and rewards that the agent encounters during training.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T00:38:31.780Z
Answers.Text: It's not fully clear to me this part "You have two main choices for collecting data ...": so are you suggesting to "collect data" in the sense that the OP should really store a dataset first and then replay that dataset to perform SARSA somehow offline? Or are you suggesting to "collect data", but you're actually referring to **where** SARSA can be trained (i.e. gain experience)? It seems more the second, but the suggestion would be applicable to both cases, anyway.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T07:39:03.973Z
Answers.Text: @nbro: "Collecting data" is not meant to imply a dataset or offline training. I think I make clear the lack of labelling and duration of useful storage in the previous paragraph. However, there is still data involved in the proceess and it has to be acquired somehow. "Collect" also has the meaning of "fetch" (as in "I collected the kids from school"). I have changed to more neutral "acquire", but open to suggestions
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-06T08:05:40.053Z
Answers.LastActivityDate: 2021-01-08T07:39:35.650Z
Answers.LastEditDate: 2021-01-08T07:39:35.650Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am currently working on my master's thesis and going to apply <strong>Deep-SARSA</strong> as my DRL algorithm. The problem is that there is <strong>no datasets available</strong> and I guess that I should generate them somehow. <strong>Datasets generation</strong> seems a common feature in this specific subject as stated in [<a href="https://ieeexplore.ieee.org/document/9039672" rel="nofollow noreferrer">1</a>]</p>
  <blockquote>
  <p>When a dataset is not available, learning is performed through experience.</p>
  </blockquote>
  <p>I am wondering <strong>how to generate datasets when the environment is not as simple as a tic-tac-toe</strong> or a maze problem and <strong>what the experience means</strong>.</p>
  <p>PS: The environment consists of 15 mobile users and 3 edge servers, each of which covers a number of mobile users. Each mobile user might generate a computationally heavy-load task and at the beginning of each timestep and can process the task itself or requests its associated edge server to do the processing. If the associated edge server is not capable of processing, due to some reasons, it requests a nearby edge server to lend it a hand. The optimization problem (reward) is to reduce time and energy consumption (multi-objective optimization). Each server has a DRL agent that makes offloading decisions.</p>
  <p>I'd really appreciate your suggestions and help.</p>
  
Tags: <reinforcement-learning><datasets><environment><sarsa><on-policy-methods>
Title: How should I generate datasets for a SARSA agent when the environment is not simple?
UserDisplayName: Shahin
ThreadText: # How should I generate datasets for a SARSA agent when the environment is not simple?
  
  (Asked by: username_0 on 2021-01-06)
  
  I am currently working on my master's thesis and going to apply **Deep-SARSA** as my DRL algorithm. The problem is that there is **no datasets available** and I guess that I should generate them somehow. **Datasets generation** seems a common feature in this specific subject as stated in [[1](https://ieeexplore.ieee.org/document/9039672)]
  
  > When a dataset is not available, learning is performed through experience.
  
  I am wondering **how to generate datasets when the environment is not as simple as a tic-tac-toe** or a maze problem and **what the experience means**.
  
  PS: The environment consists of 15 mobile users and 3 edge servers, each of which covers a number of mobile users. Each mobile user might generate a computationally heavy-load task and at the beginning of each timestep and can process the task itself or requests its associated edge server to do the processing. If the associated edge server is not capable of processing, due to some reasons, it requests a nearby edge server to lend it a hand. The optimization problem (reward) is to reduce time and energy consumption (multi-objective optimization). Each server has a DRL agent that makes offloading decisions.
  
  I'd really appreciate your suggestions and help.
  
  **Tags:** reinforcement-learning, datasets, environment, sarsa, on-policy-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-06. Score: 4 | Accepted answer
  
  > I am wondering **how to generate datasets when the environment is not as simple as a tic-tac-toe** or a maze problem
  
  There is no difference in concept, which is why tic-tac-toe and maze problems are used to teach.
  
  As you have noted, the main difference between reinforcement learning (RL) and supervised learning is that RL does not use labeled datasets. If you are using SARSA then you would not expect to use any record of previous experience either because SARSA is designed to work on-policy and online - which means that data needs to be generated *during* training. Training data for SARSA is typically stored only temporarily before being used, or is used immediately (you might keep a log of it for analysis or to help document your thesis, but that log will not be used for further training by the agent). This is different to Q-learning and DQN, which could in theory make use of longer-term stored experience.
  
  You have two main choices for acquiring data:
  
  * Use a real environment. In your case, set up 15 mobile users and 3 edge servers. Instrument the environment to collect state and reward data for the agent. Implement the agent as the real decision maker in this environment.
  * Simulate the environment. Write a simulation that models user behaviour and server loading. Instrument that to provide state and reward data, and integrate your learning agent with it. Typically the agent will call the environment's `step` function, passing the action choice as an argument and receiving reward and state data back.
  
  If you can simulate the environment, this is likely to be preferable to you since you will likely use less compute resources (than 3 servers and 15 mobile phones) and can run the training faster than real time. Deep reinforcement learning can use a large amount of experience to converge on near-optimal policies, and fast simulations can help because they generate experience faster than reality.
  
  You can also do both approaches. Train an initial agent in simulation, then implement the real version once it reaches a good level of performance in simulation. You can even have the agent continue to learn and refine behaviour in production. Given that you are working with SARSA, this may be an important part of the intent of your project, that the agent continues to adapt to changes in user behaviour and server load over time. In fact this is a key advantage of SARSA over Q-learning, that it should be more reliable and safe to use in such a continuous learning scenario deployed to production.
  
  > and **what the experience means**.
  
  The experience in reinforcement learning is the record of states, actions and rewards that the agent encounters during training.
  
  > **Comments:**
  > **username_2**: It's not fully clear to me this part "You have two main choices for collecting data ...": so are you suggesting to "collect data" in the sense that the OP should really store a dataset first and then replay that dataset to perform SARSA somehow offline? Or are you suggesting to "collect data", but you're actually referring to **where** SARSA can be trained (i.e. gain experience)? It seems more the second, but the suggestion would be applicable to both cases, anyway.  
  > **username_1**: @username_2: "Collecting data" is not meant to imply a dataset or offline training. I think I make clear the lack of labelling and duration of useful storage in the previous paragraph. However, there is still data involved in the proceess and it has to be acquired somehow. "Collect" also has the meaning of "fetch" (as in "I collected the kids from school"). I have changed to more neutral "acquire", but open to suggestions  
  
  ---
  
  