Answers.Body: <p>In the question, you are not describing the environment <em>changing</em>. Instead, there is a fixed 20% chance of a bad weather event each year. Such events can me modelled as a static environment with stochastic results.*</p>
  <p>If nothing else happened in the year, it is easy to calculate the <em>expected</em> immediate reward for each action:</p>
  <p><em>Not planting seeds.</em> <span class="math-container">$0$</span></p>
  <p><em>Planting seeds.</em> <span class="math-container">$-1000 + 0.8 \times 2000 = 600$</span></p>
  <p>So the question resolves to: Can a learning algorithm discover these odds, and make the correct decision, based only on its experience of individual years? Perhaps it will be unlucky in its first year and simply give up?</p>
  <p>The answer is yes, most reinforcement learning (RL) algorithms will cope just fine, and learn that planting seeds is the best option in the simple environment that you have proposed.</p>
  <p>There is an important caveat: To learn the true value of each option, algorithm must be one that <em>explores</em>. That means, whichever action the agent thinks is currently best, there must be some chance that it decides to take (what it thinks is) a worse option. If the worse option turns out better than it expected, the agent can adjust its expectations, and maybe choose that one more frequently in future, and vice versa if the option turns out worse than expected.</p>
  <p>Nearly all RL methods do this, by setting some kind of exploration parameter. The more the agent explores, the more it will learn about alternatives to its current best guess at optimal behaviour. However, whilst exploring it may pay the price for making many sub-optimal decisions. Your planning agent may decide to not plant anything every few years, just to see if that is better, even after it has established that planting is the best way to get profit.</p>
  <p>It is often hard to balance between the learning obtained by trying different options vs doing what the agent currently thinks is best. This is called the <a href="https://conceptually.org/concepts/explore-or-exploit" rel="nofollow noreferrer">exploration-exploitation tradeoff</a>, and is a fundamental issue in RL.</p>
  <hr />
  <p>* For comparison, a non-stationary environment in RL would be one in which the probabilities and/or reward distributions change so that values learned from older experience are no longer valid. If climate change impacted your grape-growing region such that after some year the chance of a bad storm was 80%, and the agent had no way of knowing that in advance, then the agent might lose money trying to follow its previously best policy. If it was still learning, it would adapt to these new rules eventually though.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-14T02:11:48.973Z
Answers.Text: Ok, so suppose the weather changes not by chance, but by motivation. There is some superhuman or other entity that might want to alter the weather such that the grapes die. Can the agent learn to adapt?
Answers.UserDisplayName: Montana Burr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-14T07:53:26.380Z
Answers.Text: @MontanaBurr: That depends on what you mean by "motivation". You need to be clear in your definitions if you want an answer. Most importantly, is the "motivation" based on some non-random factor, and can the agent observe or predict that factor? This is quite different from your original question, needs clarification, and there could be lots of detail in any answer, so I suggest you ask a new question about that.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-13T19:48:41.437Z
Answers.LastActivityDate: 2022-01-13T23:10:20.707Z
Answers.LastEditDate: 2022-01-13T23:10:20.707Z
Answers.UserDisplayName: Neil Slater
Body: <p>I've been reading about reinforcement learning, and it seems to me that reinforcement learning assumes the environment is static, and therefore that the reward for taking a particular action will be the same from time to time. But what if something happens outside the agent's control, that changes the reward?</p>
  <p>As a simplified example, suppose the agent in a RL algorithm is a grape farmer living on a planet known for extreme weather events. Its goal is to maximize profit/minimize losses. Grape seeds cost 1000 dollars. It takes 1 year for the grape seeds to grow delicious grapes. There is a 20% chance of the weather destroying the grapes during the year, leading to the agent losing 1000 dollars. However, if the weather does NOT destroy the grapes, the agent wins $2000.</p>
  <p>The agent starts the year with two choices:</p>
  <ul>
  <li>Grow grapes</li>
  <li>Don't grow grapes</li>
  </ul>
  <p>In its first year, an adverse weather event occurs, killing the grapes. Will the agent ever grow grapes again, or will it forever decide that the best way to win is to not plant grapes? Likewise, if it ever DOES grow grapes, can we trust it to know when not to grow grapes?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-13T19:57:20.580Z
Comments.Text: Why do you say the weather is external to the environment? Your environment is the weather.
Comments.UserDisplayName: Brian O'Donnell
Tags: <reinforcement-learning>
Title: Does reinforcement learning lend itself well to changes in the environment due to external factors?
UserDisplayName: Montana Burr
ThreadText: # Does reinforcement learning lend itself well to changes in the environment due to external factors?
  
  (Asked by: username_0 on 2022-01-13)
  
  I've been reading about reinforcement learning, and it seems to me that reinforcement learning assumes the environment is static, and therefore that the reward for taking a particular action will be the same from time to time. But what if something happens outside the agent's control, that changes the reward?
  
  As a simplified example, suppose the agent in a RL algorithm is a grape farmer living on a planet known for extreme weather events. Its goal is to maximize profit/minimize losses. Grape seeds cost 1000 dollars. It takes 1 year for the grape seeds to grow delicious grapes. There is a 20% chance of the weather destroying the grapes during the year, leading to the agent losing 1000 dollars. However, if the weather does NOT destroy the grapes, the agent wins $2000.
  
  The agent starts the year with two choices:
  
  * Grow grapes
  * Don't grow grapes
  
  In its first year, an adverse weather event occurs, killing the grapes. Will the agent ever grow grapes again, or will it forever decide that the best way to win is to not plant grapes? Likewise, if it ever DOES grow grapes, can we trust it to know when not to grow grapes?
  
  **Tags:** reinforcement-learning
  
  > **Comments:**
  > **username_1**: Why do you say the weather is external to the environment? Your environment is the weather.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-13. Score: 1
  
  In the question, you are not describing the environment *changing*. Instead, there is a fixed 20% chance of a bad weather event each year. Such events can me modelled as a static environment with stochastic results.\*
  
  If nothing else happened in the year, it is easy to calculate the *expected* immediate reward for each action:
  
  *Not planting seeds.* $0$
  
  *Planting seeds.* $-1000 + 0.8 \times 2000 = 600$
  
  So the question resolves to: Can a learning algorithm discover these odds, and make the correct decision, based only on its experience of individual years? Perhaps it will be unlucky in its first year and simply give up?
  
  The answer is yes, most reinforcement learning (RL) algorithms will cope just fine, and learn that planting seeds is the best option in the simple environment that you have proposed.
  
  There is an important caveat: To learn the true value of each option, algorithm must be one that *explores*. That means, whichever action the agent thinks is currently best, there must be some chance that it decides to take (what it thinks is) a worse option. If the worse option turns out better than it expected, the agent can adjust its expectations, and maybe choose that one more frequently in future, and vice versa if the option turns out worse than expected.
  
  Nearly all RL methods do this, by setting some kind of exploration parameter. The more the agent explores, the more it will learn about alternatives to its current best guess at optimal behaviour. However, whilst exploring it may pay the price for making many sub-optimal decisions. Your planning agent may decide to not plant anything every few years, just to see if that is better, even after it has established that planting is the best way to get profit.
  
  It is often hard to balance between the learning obtained by trying different options vs doing what the agent currently thinks is best. This is called the [exploration-exploitation tradeoff](https://conceptually.org/concepts/explore-or-exploit), and is a fundamental issue in RL.
  
  ---
  
  \* For comparison, a non-stationary environment in RL would be one in which the probabilities and/or reward distributions change so that values learned from older experience are no longer valid. If climate change impacted your grape-growing region such that after some year the chance of a bad storm was 80%, and the agent had no way of knowing that in advance, then the agent might lose money trying to follow its previously best policy. If it was still learning, it would adapt to these new rules eventually though.
  
  > **Comments:**
  > **username_0**: Ok, so suppose the weather changes not by chance, but by motivation. There is some superhuman or other entity that might want to alter the weather such that the grapes die. Can the agent learn to adapt?  
  > **username_2**: @username_0: That depends on what you mean by "motivation". You need to be clear in your definitions if you want an answer. Most importantly, is the "motivation" based on some non-random factor, and can the agent observe or predict that factor? This is quite different from your original question, needs clarification, and there could be lots of detail in any answer, so I suggest you ask a new question about that.  
  
  ---
  
  