Answers.Body: <p>This depends on the complexity of the environment being learned, and the purpose for learning it. There is no general answer.</p>
  
  <p>For the simple environments used to teach reinforcement learning (RL), often the optimal solution is obvious, or can be calculated and proven optimal. For instance, any environment that can be solved using policy iteration will have a known optimal policy and optimal value function. The goals of these environments are to teach, or to check correctness of agents - it helps in these cases to have a well known correct answer.</p>
  
  <p>At the next level up in terms of complexity are well-studied environments which can have achievable targets set for learning agents. The goals of these environments include getting useful metrics for learning agents, such as how many episodes it takes a particular implementation to learn it well enough. Defining "well enough" is a matter of experience with existing agents. </p>
  
  <p>Getting more complex still, in general it is not possible to know whether an agent has fully optimised against its environment. The subject area of sequential decision making that includes RL agents can cover scenarios such as driving a car or playing a computer game. We don't know when any agent, whether they are based on RL or some other approach, has fully learning an environment, and instead must construct tests of behaviour - e.g. make an agent simulate driving in a set of scenarios, and expect at least safe behaviour in each of them, essentially a driving test similar to one a person might take. In these environments, often the tests are based on "good enough to use" goals. We can say an agent has learned to drive if it drives more safely than an average human.</p>
  
  <p>In the special case of competitive games, we can score agents against each other or against human players. You might say that an agent has learned its environment if it beats some standard player, but also you can rank agents against each other and declare a particular agent as the current best.</p>
  
  <p>It is possible to mix and match these ideas. The Atari games learning suite <a href="https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark" rel="nofollow noreferrer">has benchmark scores to reach</a> that count as "standard human", and recently agents have been published that beat all of those scores. </p>
  
  <blockquote>
    <p>What happens if you continue training after the agent has learnt the environment? Will it perform by reaching its goal every time or will there be failed episodes?</p>
  </blockquote>
  
  <p>If you include training episodes, then RL learns mainly by "trial and error". So you should expect an agent to make <em>deliberate</em> mistakes as it tests to see what happens. In some environments these could be critical mistakes leading to failed episodes.</p>
  
  <p>If you ignore the training episodes and are interested only in performance without exploration - e.g. testing every few hundred episodes - then you can expect performance to vary depending on the type of agent and environment. Some agents even exhibit "catastrophic forgetting", which as the name implies causes performance to drop significantly - this can be caused by a successful agent over-fitting to all the recent successful episodes without errors it just experienced, and losing the ability to predict the true lower value of incorrect actions.</p>
  
  <p>Neither failed episodes during training nor catastrophic forgetting are inevitable. It depends on the environment and type of agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T13:23:44.850Z
Answers.Text: Thank you for the detailed answer. Please tell me what is the difference between training and testing (aka evaluation if I understood correctly)? As I see it, both imply the same procedure: select an action, apply to the environment, get reward and state...
Answers.UserDisplayName: Cristian M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T14:00:06.300Z
Answers.Text: @CristianM: During training the agent will - with some probability - deliberately take actions that it currently evaluates as not optimal, in order to refine its understanding of them (maybe its estimates are wrong, it can only find out by trying). This can reduce its performance. That is what I am referring to in the paragraph starting "If you include training episodes . . ."
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T14:32:56.270Z
Answers.Text: OK. I am referring in a general sense. I will post a separate question for this.
Answers.UserDisplayName: Cristian M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T12:09:56.493Z
Answers.LastActivityDate: 2020-05-04T12:09:56.493Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm new to reinforcement learning and trying to understand it. </p>
  
  <p>If you train an agent using a reinforcement learning algorithm (discrete or continuous) on an environment (real or simulated), then how do you know if the agent has learnt its environment? Should it reach its goal on every run (episode)? (Any literature references are also welcome)</p>
  
  <p>Is this related to the reward threshold defined in the environment?</p>
  
  <p>What happens if you continue training after the agent has learnt the environment? Will it perform by reaching its goal every time or will there be failed episodes?</p>
  
Tags: <reinforcement-learning><rewards><intelligent-agent><learning-algorithms><environment>
Title: How do you know if an agent has learnt its environment in reinforcement learning?
UserDisplayName: Cristian M
ThreadText: # How do you know if an agent has learnt its environment in reinforcement learning?
  
  (Asked by: username_0 on 2020-05-04)
  
  I'm new to reinforcement learning and trying to understand it.
  
  If you train an agent using a reinforcement learning algorithm (discrete or continuous) on an environment (real or simulated), then how do you know if the agent has learnt its environment? Should it reach its goal on every run (episode)? (Any literature references are also welcome)
  
  Is this related to the reward threshold defined in the environment?
  
  What happens if you continue training after the agent has learnt the environment? Will it perform by reaching its goal every time or will there be failed episodes?
  
  **Tags:** reinforcement-learning, rewards, intelligent-agent, learning-algorithms, environment
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-04. Score: 0 | Accepted answer
  
  This depends on the complexity of the environment being learned, and the purpose for learning it. There is no general answer.
  
  For the simple environments used to teach reinforcement learning (RL), often the optimal solution is obvious, or can be calculated and proven optimal. For instance, any environment that can be solved using policy iteration will have a known optimal policy and optimal value function. The goals of these environments are to teach, or to check correctness of agents - it helps in these cases to have a well known correct answer.
  
  At the next level up in terms of complexity are well-studied environments which can have achievable targets set for learning agents. The goals of these environments include getting useful metrics for learning agents, such as how many episodes it takes a particular implementation to learn it well enough. Defining "well enough" is a matter of experience with existing agents.
  
  Getting more complex still, in general it is not possible to know whether an agent has fully optimised against its environment. The subject area of sequential decision making that includes RL agents can cover scenarios such as driving a car or playing a computer game. We don't know when any agent, whether they are based on RL or some other approach, has fully learning an environment, and instead must construct tests of behaviour - e.g. make an agent simulate driving in a set of scenarios, and expect at least safe behaviour in each of them, essentially a driving test similar to one a person might take. In these environments, often the tests are based on "good enough to use" goals. We can say an agent has learned to drive if it drives more safely than an average human.
  
  In the special case of competitive games, we can score agents against each other or against human players. You might say that an agent has learned its environment if it beats some standard player, but also you can rank agents against each other and declare a particular agent as the current best.
  
  It is possible to mix and match these ideas. The Atari games learning suite [has benchmark scores to reach](https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark) that count as "standard human", and recently agents have been published that beat all of those scores.
  
  > What happens if you continue training after the agent has learnt the environment? Will it perform by reaching its goal every time or will there be failed episodes?
  
  If you include training episodes, then RL learns mainly by "trial and error". So you should expect an agent to make *deliberate* mistakes as it tests to see what happens. In some environments these could be critical mistakes leading to failed episodes.
  
  If you ignore the training episodes and are interested only in performance without exploration - e.g. testing every few hundred episodes - then you can expect performance to vary depending on the type of agent and environment. Some agents even exhibit "catastrophic forgetting", which as the name implies causes performance to drop significantly - this can be caused by a successful agent over-fitting to all the recent successful episodes without errors it just experienced, and losing the ability to predict the true lower value of incorrect actions.
  
  Neither failed episodes during training nor catastrophic forgetting are inevitable. It depends on the environment and type of agent.
  
  > **Comments:**
  > **username_0**: Thank you for the detailed answer. Please tell me what is the difference between training and testing (aka evaluation if I understood correctly)? As I see it, both imply the same procedure: select an action, apply to the environment, get reward and state...  
  > **username_1**: @username_0: During training the agent will - with some probability - deliberately take actions that it currently evaluates as not optimal, in order to refine its understanding of them (maybe its estimates are wrong, it can only find out by trying). This can reduce its performance. That is what I am referring to in the paragraph starting "If you include training episodes . . ."  
  > **username_0**: OK. I am referring in a general sense. I will post a separate question for this.  
  
  ---
  
  