Answers.Body: <h4>Short Intro</h4>
  <p>It's very common for people to think that Deep Learning is a &quot;superior form&quot; of Neural Network, a &quot;smarter model&quot;. And then they try to use DL for solving simple tasks and they'll find more problems than solutions.</p>
  <blockquote>
  <p>We might think of plane as superior to a bike. But when we need to buy some bread for breakfast, taking a plane is not even an overkill. It's just nonsense.</p>
  </blockquote>
  <h3>Your problem</h3>
  <p>In a similar way, I've seen people thinking <strong>Reinforcement Learning</strong> as somehow superior to Supervised or Unsupervised Learning. So let's establish a baseline:</p>
  <ul>
  <li>The choice between Reinforcement Learning or Supervised Learning is <em>not</em> about superiority, but rather the nature of the task and your available dataset.</li>
  </ul>
  <h2>Reinforcement Learning</h2>
  <p>Great when your problem can be modeled as an agent interacting with an environment. The agent will sense (input), process (policy) and act (output). The policy is learned based on the reward of each action.</p>
  <p>In most RL tasks (like strategy games), you can't objectively rate each individual action. Instead, it takes multiple actions before the outcome is obtained. But you can't train your model when you can't measure it's performance. So how to train a RL model?</p>
  <h4>Policy:</h4>
  <p>A policy is like a function that maps states into actions. When you can't measure the performance of each individual action, you create a policy.</p>
  <p>You let the agent interact with the environment (play the game) until you have a score. If the environment is stochastic (the game depends on luck), you might want several rollouts (play lots of times) before evaluating the policy.</p>
  <p>So, for acquiring a single data sample, you might need to let an agent to play a game several times.</p>
  <p>But if you can directly measure the agent's perform every each action, congratulations! You can probably save lots of computational power by modeling the problem as a:</p>
  <h2>Supervised Learning</h2>
  <p>The model is learned based on the input-output pairs. A training dataset.</p>
  <h3>LSTM</h3>
  <p>Great when your input is sequential and your output is a function of the previous state.</p>
  <h3>Random Forest</h3>
  <p>Great when your input is not sequential, but you have a lot of features.</p>
  <h2>Unsupervised Learning</h2>
  <p>Great when your input is not sequential and you want to discover hidden structure in your data.</p>
  <p>I hope this helps you with both questions. :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T16:08:25.603Z
Answers.Text: Thank you for the insightful categorization. Since my data is sequential, is it correct to say that if I know the best policy for every predicted future ( Let's say coming from LSTM), that would be the same as RL predicting the future and finding the best policy? How important is it that the agent in RL is able to affect the environment?
Answers.UserDisplayName: user101464
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T18:33:46.320Z
Answers.Text: I've edited the answer, adding brief description about policy. I hope it answers your question.
Answers.UserDisplayName: Andre Goulart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T20:31:45.203Z
Answers.Text: Thank you, Andre Goulart. It was a very informative answer.
Answers.UserDisplayName: user101464
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T15:29:08.033Z
Answers.LastActivityDate: 2021-08-31T18:32:36.933Z
Answers.LastEditDate: 2021-08-31T18:32:36.933Z
Answers.UserDisplayName: Andre Goulart
Body: <p>I am trying to get my head around a problem where the action by the agent can not change the environment. Without going into details, my problem is about error correction in an stochastic environment.</p>
  <p>So, here the action by agent can not change the environment that causes these error and all we can do is to smartly correct as the errors happen. I am currently thinking about using Reinforcement learning for this agent who could correct the errors.</p>
  <p>Now my questions are:</p>
  <ol>
  <li>Would reinforcing learning be an overkill since the agent can not  influence the environment?</li>
  <li>How do RL, LSTM, and even random forest compare in such scenarios?</li>
  </ol>
  <p>Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T06:53:15.427Z
Comments.Text: Could you clarify what the action is to "smartly correct"? Is it a numerical value (or vector of values) that adds to some other prediction, or is it a selection from different types of correction (e.g. choose between method "A" vs method "B" for correction)?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T14:58:40.630Z
Comments.Text: @NeilSlater. Here, by "smartly correct" I mean, if my machine learning approach (whatever it is which is not implemented yet) is successful and it comes up with a strategy then it can "smartly correct" the detrimental effect of the noise. And it is an action. for example, if noise flips a binary bit, the agent should be able to fix that by flipping it again.
Comments.UserDisplayName: user101464
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T15:12:20.817Z
Comments.Text: Sorry I don't understand that. Could you please state what form your actions take? Are the actions numerical (e.g. "correct by +1.345") or categorical (e.g. "correct using choice 5 out of 10")? Or is the action literally a vector of bits to flip? I may have further questions to clarify details of your reward/loss function and how you are establishing ground truth, but best to start with details of action. What I am trying to understand is what the supervised learning version of this would look like, in order to figure out the asnwer to your second question.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T15:53:29.710Z
Comments.Text: Thanks@NeilSlater  The action is one of few operations the agent can take, let's say, integer from 1 to 4. For example, 1 is for turning on switch 1, and so on. To be more specific, I few sensors that give me some values, like temperature, how crowded the area is , and so on. Now I would like to find the best strategy, which is turning on and off some switches based on the information I got from sensors to correct the disturbances caused by the environment. As it is obvious, I can not do anything to change the environment that causing these errors, I can not even reduce them.
Comments.UserDisplayName: user101464
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T17:12:01.550Z
Comments.Text: OK, so your error correction choice is discrete, one of a few options. How do you then know whether the action is correct? Do you have some ground truth for historical data? Do you have any way to assess an action taken in either simulation or in reality as being the correct action?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T20:29:23.273Z
Comments.Text: The party who creates that system knows what state the system is supposed to be at all time. In ordinary operation that knowledge is hidden from us and all we can do is just keep correcting for errors cause by the environment. However, in training we can start with a state that we create ourselves and have perfect knowledge and the RL agent is supposed to preserve it by error correction. So, creating training set is easy.
Comments.UserDisplayName: user101464
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-01T07:07:23.760Z
Comments.Text: I am interested in your use of the word "state" here. You imply that there is hidden state, and that the agent's actions do in fact impact it? Could you clarify what the agent's actions do? The implied behaviour from your question is that the environment produces some data output, depending on a known state, which is consumed outside of the environment, but the output can be wrong (again depending on the known state) and the agent's job is to correct that output. Your comment above implies instead there is hidden state and it is the agent's job to somehow maintain it?
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><long-short-term-memory><random-forests>
Title: Reinforcing Learning when action has no effect on the environment
UserDisplayName: user101464
ThreadText: # Reinforcing Learning when action has no effect on the environment
  
  (Asked by: username_0 on 2021-08-31)
  
  I am trying to get my head around a problem where the action by the agent can not change the environment. Without going into details, my problem is about error correction in an stochastic environment.
  
  So, here the action by agent can not change the environment that causes these error and all we can do is to smartly correct as the errors happen. I am currently thinking about using Reinforcement learning for this agent who could correct the errors.
  
  Now my questions are:
  
  1. Would reinforcing learning be an overkill since the agent can not influence the environment?
  2. How do RL, LSTM, and even random forest compare in such scenarios?
  
  Thank you.
  
  **Tags:** reinforcement-learning, long-short-term-memory, random-forests
  > **Comments:**
  > **username_1**: Could you clarify what the action is to "smartly correct"? Is it a numerical value (or vector of values) that adds to some other prediction, or is it a selection from different types of correction (e.g. choose between method "A" vs method "B" for correction)?  
  > **username_0**: @NeilSlater. Here, by "smartly correct" I mean, if my machine learning approach (whatever it is which is not implemented yet) is successful and it comes up with a strategy then it can "smartly correct" the detrimental effect of the noise. And it is an action. for example, if noise flips a binary bit, the agent should be able to fix that by flipping it again.  
  > **username_1**: Sorry I don't understand that. Could you please state what form your actions take? Are the actions numerical (e.g. "correct by +1.345") or categorical (e.g. "correct using choice 5 out of 10")? Or is the action literally a vector of bits to flip? I may have further questions to clarify details of your reward/loss function and how you are establishing ground truth, but best to start with details of action. What I am trying to understand is what the supervised learning version of this would look like, in order to figure out the asnwer to your second question.  
  > **username_0**: Thanks@NeilSlater  The action is one of few operations the agent can take, let's say, integer from 1 to 4. For example, 1 is for turning on switch 1, and so on. To be more specific, I few sensors that give me some values, like temperature, how crowded the area is , and so on. Now I would like to find the best strategy, which is turning on and off some switches based on the information I got from sensors to correct the disturbances caused by the environment. As it is obvious, I can not do anything to change the environment that causing these errors, I can not even reduce them.  
  > **username_1**: OK, so your error correction choice is discrete, one of a few options. How do you then know whether the action is correct? Do you have some ground truth for historical data? Do you have any way to assess an action taken in either simulation or in reality as being the correct action?  
  > **username_0**: The party who creates that system knows what state the system is supposed to be at all time. In ordinary operation that knowledge is hidden from us and all we can do is just keep correcting for errors cause by the environment. However, in training we can start with a state that we create ourselves and have perfect knowledge and the RL agent is supposed to preserve it by error correction. So, creating training set is easy.  
  > **username_1**: I am interested in your use of the word "state" here. You imply that there is hidden state, and that the agent's actions do in fact impact it? Could you clarify what the agent's actions do? The implied behaviour from your question is that the environment produces some data output, depending on a known state, which is consumed outside of the environment, but the output can be wrong (again depending on the known state) and the agent's job is to correct that output. Your comment above implies instead there is hidden state and it is the agent's job to somehow maintain it?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-31. Score: 2
  
  #### Short Intro
  
  It's very common for people to think that Deep Learning is a "superior form" of Neural Network, a "smarter model". And then they try to use DL for solving simple tasks and they'll find more problems than solutions.
  
  > We might think of plane as superior to a bike. But when we need to buy some bread for breakfast, taking a plane is not even an overkill. It's just nonsense.
  
  ### Your problem
  
  In a similar way, I've seen people thinking **Reinforcement Learning** as somehow superior to Supervised or Unsupervised Learning. So let's establish a baseline:
  
  * The choice between Reinforcement Learning or Supervised Learning is *not* about superiority, but rather the nature of the task and your available dataset.
  
  ## Reinforcement Learning
  
  Great when your problem can be modeled as an agent interacting with an environment. The agent will sense (input), process (policy) and act (output). The policy is learned based on the reward of each action.
  
  In most RL tasks (like strategy games), you can't objectively rate each individual action. Instead, it takes multiple actions before the outcome is obtained. But you can't train your model when you can't measure it's performance. So how to train a RL model?
  
  #### Policy:
  
  A policy is like a function that maps states into actions. When you can't measure the performance of each individual action, you create a policy.
  
  You let the agent interact with the environment (play the game) until you have a score. If the environment is stochastic (the game depends on luck), you might want several rollouts (play lots of times) before evaluating the policy.
  
  So, for acquiring a single data sample, you might need to let an agent to play a game several times.
  
  But if you can directly measure the agent's perform every each action, congratulations! You can probably save lots of computational power by modeling the problem as a:
  
  ## Supervised Learning
  
  The model is learned based on the input-output pairs. A training dataset.
  
  ### LSTM
  
  Great when your input is sequential and your output is a function of the previous state.
  
  ### Random Forest
  
  Great when your input is not sequential, but you have a lot of features.
  
  ## Unsupervised Learning
  
  Great when your input is not sequential and you want to discover hidden structure in your data.
  
  I hope this helps you with both questions. :)
  
  > **Comments:**
  > **username_0**: Thank you for the insightful categorization. Since my data is sequential, is it correct to say that if I know the best policy for every predicted future ( Let's say coming from LSTM), that would be the same as RL predicting the future and finding the best policy? How important is it that the agent in RL is able to affect the environment?  
  > **username_2**: I've edited the answer, adding brief description about policy. I hope it answers your question.  
  > **username_0**: Thank you, Andre Goulart. It was a very informative answer.  
  
  ---
  
  