Answers.Body: <p>Overfitting refers to a model being stuck in a local minimum while trying to minimise a loss function. In Reinforcement Learning the aim is to learn an optimal policy by maximising or minimising a non-stationary objective-function which depends on the action policy, so overfitting is not exactly like in the supervised scenario, but you can definitely talk about <em>sub-optimal policies</em>.</p>
  <p>If we think of a specific task like avoiding stationary objects, a simple sub-optimal policy would be to just stay still without moving at all, or moving in circles if the reward function was designed to penalise lack of movements.</p>
  <p>The way to avoid an agent to learn sub-optimal policies is to find a good compromise between <em>exploitation</em>, i.e. the constant selection of the next action to take based on the maximum expected reward possible, and <em>exploration</em>, i.e. a random selection of the next action to take regardless of the rewards. Here's a link to an introduction to the topic:
  <a href="https://www.manifold.ai/exploration-vs-exploitation-in-reinforcement-learning" rel="nofollow noreferrer">Exploration and Exploitation in Reinforcement Learning</a></p>
  <p>It is worth mentioning that sometimes an agent can actually outsmart humans though, some examples are reported in this paper <a href="https://arxiv.org/pdf/1803.03453.pdf" rel="nofollow noreferrer">The Surprising Creativity of Digital Evolution</a>. I particularly like the story of the insect agent trained to learn to walk while minimising the contact with the floor surface. The agent surprisingly managed to learn to walk without touching the ground at all. When the authors checked what was going on they discovered that the insect leaned to flip itself and then walk using its fake 'elbows' (fig7 in the linked paper). I add this story just to point out that most of the time the design of the reward function is itself even more important than exploration and exploitation tuning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T16:33:44.977Z
Answers.Text: I wouldn't describe overfitting as the situation where a model gets stuck in a local minimum, but as the "gap between training and test performances". This definition should also make it clearer why it _may_ not make sense to talk about overfitting in RL. If we use your description of overfitting, one could argue that "sub-optimal policies" correspond to local minima, and they wouldn't be wrong. I think it would also be useful you if commented the plots that the asker provided. What do thee plots represent and can we understand whether the RL agent is "overfitting" from them?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T06:02:52.983Z
Answers.Text: ah... I see.. so can you help me to understand why my Agent's Score decreasing after 400 episodes? it keeps decreasing until almost 0 after 1000 episodes. Because in this game the reward is the score (0 to 12), so I think the problem is not a sub-optimal policy, right?
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T13:23:47.067Z
Answers.Text: @malioboro Could you explain a bit more in details the game and what reward function you're using? Honestly the only thing I can tell from your graphs is that after 200 iterations your neural net starts overfitting, affecting the performances of the agent in selecting the next action. You could just stop training the deep net when the loss start increase, and keep using the same model to train the policy of your agent. Also, consider that sometimes it can take more than 1k, even 10k iterations to reach an optimal policy, the drop you see might be just a random fluctuation.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T13:40:01.187Z
Answers.Text: sorry about the last sentence, I read the comment quickly, if after 1k iterations the reward approach zero then I think the problem is most definitely related to the performance of the neural net.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T14:08:48.550Z
Answers.Text: @EdoardoGuerriero thank you so much for your input, I solve this problem by tune the RL model to use more steps before update the target network, and also tune the NN parameters as you said
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T14:09:47.503Z
Answers.Text: and that's true that RL resuls is very fluctuative, some times the score is decreasing but after few episodes it starts increasing again
Answers.UserDisplayName: malioboro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T19:12:44.917Z
Answers.Text: In Reinforcement Learning you do want to minimise/maximise a loss function!
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T14:53:07.467Z
Answers.Text: @RémyHosseinkhanBoucher not necessarily. Classic q learning for example require only a reward function, which is much different to a loss function conceptually.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T16:18:38.293Z
Answers.LastActivityDate: 2022-11-28T14:50:06.097Z
Answers.LastEditDate: 2022-11-28T14:50:06.097Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p><a href="https://ai.stackexchange.com/a/20135/2444">The accepted answer</a> does not provide a good definition of <em>over-fitting</em>, which actually exists and is a defined concept in reinforcement learning too. For example, the paper <a href="http://proceedings.mlr.press/v97/cobbe19a/cobbe19a.pdf" rel="noreferrer">Quantifying Generalization in Reinforcement Learning</a> completely focuses on this issue. Let me give you more details.</p>
  <h3>Over-fitting in supervised learning</h3>
  <p>In <strong>supervised learning (SL)</strong>, <strong>over-fitting</strong> is defined as the difference (or gap) in the performance of the ML model (such as a neural network) on the training and test datasets. If the model performs significantly better on the training dataset than on the test dataset, then the ML model has over-fitted the training data. Consequently, it has not generalized (well enough) to other data other than the training data (i.e. the test data). The relationship between over-fitting and <a href="https://ai.stackexchange.com/a/16737/2444">generalization</a> should now be clearer.</p>
  <h3>Over-fitting in reinforcement learning</h3>
  <p>In <strong>reinforcement learning (RL)</strong> (you can find a brief recap of what RL is <a href="https://ai.stackexchange.com/a/20957/2444">here</a>), you want to find an <strong>optimal policy</strong> or value function (from which the policy can be derived), which can be represented by a neural network (or another model). A policy <span class="math-container">$\pi$</span> is optimal in environment <span class="math-container">$E$</span> if it leads to the highest cumulative reward in the long run in that environment <span class="math-container">$E$</span>, which is often mathematically modelled as a (partially or fully observable) Markov decision process.</p>
  <p>In some cases, you are also interested in knowing whether your policy <span class="math-container">$\pi$</span> can also be used in a different environment than the environment it has been trained in, i.e. you're interested in knowing if the knowledge acquired in that training environment <span class="math-container">$E$</span> can be <strong>transferred</strong> to a different (but typically related) environment (or task) <span class="math-container">$E'$</span>. For example, you may only be able to train your policy in a simulated environment (because of resource/safety constraints), then you want to transfer this learned policy to the real world. In those cases, you can define the concept of <strong>over-fitting</strong> in a similar way to the way we define over-fitting in SL. The only difference may be that you may say that the learned policy <em>has over-fitted the training environment</em> (rather than saying that the ML model has over-fitted the training dataset), but, given that the environment provides the <em>data</em>, then you could even say in RL that your policy has over-fitted the training data.</p>
  <h3>Catastrophic forgetting</h3>
  <p>There is also the issue of <a href="https://ai.stackexchange.com/a/13293/2444"><strong>catastrophic forgetting (CF)</strong></a> in RL, i.e., while learning, your RL agent may forget what it's previously learned, and this can even happen in the same environment. Why am I talking about CF? Because what it's happening to you is probably CF, i.e., <em>while learning</em>, the agent performs well for a while, then its performance drops (although I have read a paper that strangely defines CF differently in RL). You could also say that over-fitting is happening in your case, but, if you are continuously training and the performance changes, then CF is probably what you need to investigate. So, you should reserve the word <em>over-fitting</em> in RL when you're interested in <strong>transfer learning</strong> (i.e. the training and test environments do not coincide).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-23T12:24:31.967Z
Answers.LastActivityDate: 2020-11-23T15:38:49.910Z
Answers.LastEditDate: 2020-11-23T15:38:49.910Z
Answers.UserDisplayName: nbro
Body: <p>So this is my current result (loss and score per episode) of my RL model in a simple two players game:</p>
  <p><a href="https://i.sstatic.net/K7YTe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/K7YTe.png" alt="enter image description here" /></a></p>
  <p>I use DQN with CNN as a policy and target networks. I train my model using Adam optimizer and calculate the loss using Smooth L1 Loss.</p>
  <p>In a normal &quot;Supervised Learning&quot; situation, I can deduce that my model is overfitting. And I can imagine some methods to tackle this problem (e.g. Dropout layer, Regularization, Smaller Learning Rate, Early Stopping).</p>
  <ul>
  <li>But would that solution will also work in RL problem?</li>
  <li>Or are there any better solutions to handle overfitting in RL?</li>
  </ul>
  
Tags: <machine-learning><reinforcement-learning><overfitting>
Title: How can I handle overfitting in reinforcement learning problems?
UserDisplayName: malioboro
ThreadText: # How can I handle overfitting in reinforcement learning problems?
  
  (Asked by: username_0 on 2020-04-09)
  
  So this is my current result (loss and score per episode) of my RL model in a simple two players game:
  
  [![enter image description here](https://i.sstatic.net/K7YTe.png)](https://i.sstatic.net/K7YTe.png)
  
  I use DQN with CNN as a policy and target networks. I train my model using Adam optimizer and calculate the loss using Smooth L1 Loss.
  
  In a normal "Supervised Learning" situation, I can deduce that my model is overfitting. And I can imagine some methods to tackle this problem (e.g. Dropout layer, Regularization, Smaller Learning Rate, Early Stopping).
  
  * But would that solution will also work in RL problem?
  * Or are there any better solutions to handle overfitting in RL?
  
  **Tags:** machine-learning, reinforcement-learning, overfitting
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-04-09. Score: 4 | Accepted answer
  
  Overfitting refers to a model being stuck in a local minimum while trying to minimise a loss function. In Reinforcement Learning the aim is to learn an optimal policy by maximising or minimising a non-stationary objective-function which depends on the action policy, so overfitting is not exactly like in the supervised scenario, but you can definitely talk about *sub-optimal policies*.
  
  If we think of a specific task like avoiding stationary objects, a simple sub-optimal policy would be to just stay still without moving at all, or moving in circles if the reward function was designed to penalise lack of movements.
  
  The way to avoid an agent to learn sub-optimal policies is to find a good compromise between *exploitation*, i.e. the constant selection of the next action to take based on the maximum expected reward possible, and *exploration*, i.e. a random selection of the next action to take regardless of the rewards. Here's a link to an introduction to the topic:
  [Exploration and Exploitation in Reinforcement Learning](https://www.manifold.ai/exploration-vs-exploitation-in-reinforcement-learning)
  
  It is worth mentioning that sometimes an agent can actually outsmart humans though, some examples are reported in this paper [The Surprising Creativity of Digital Evolution](https://arxiv.org/pdf/1803.03453.pdf). I particularly like the story of the insect agent trained to learn to walk while minimising the contact with the floor surface. The agent surprisingly managed to learn to walk without touching the ground at all. When the authors checked what was going on they discovered that the insect leaned to flip itself and then walk using its fake 'elbows' (fig7 in the linked paper). I add this story just to point out that most of the time the design of the reward function is itself even more important than exploration and exploitation tuning.
  
  > **Comments:**
  > **username_2**: I wouldn't describe overfitting as the situation where a model gets stuck in a local minimum, but as the "gap between training and test performances". This definition should also make it clearer why it _may_ not make sense to talk about overfitting in RL. If we use your description of overfitting, one could argue that "sub-optimal policies" correspond to local minima, and they wouldn't be wrong. I think it would also be useful you if commented the plots that the asker provided. What do thee plots represent and can we understand whether the RL agent is "overfitting" from them?  
  > **username_0**: ah... I see.. so can you help me to understand why my Agent's Score decreasing after 400 episodes? it keeps decreasing until almost 0 after 1000 episodes. Because in this game the reward is the score (0 to 12), so I think the problem is not a sub-optimal policy, right?  
  > **username_1**: @malioboro Could you explain a bit more in details the game and what reward function you're using? Honestly the only thing I can tell from your graphs is that after 200 iterations your neural net starts overfitting, affecting the performances of the agent in selecting the next action. You could just stop training the deep net when the loss start increase, and keep using the same model to train the policy of your agent. Also, consider that sometimes it can take more than 1k, even 10k iterations to reach an optimal policy, the drop you see might be just a random fluctuation.  
  > **username_1**: sorry about the last sentence, I read the comment quickly, if after 1k iterations the reward approach zero then I think the problem is most definitely related to the performance of the neural net.  
  > **username_0**: @EdoardoGuerriero thank you so much for your input, I solve this problem by tune the RL model to use more steps before update the target network, and also tune the NN parameters as you said  
  > **username_0**: and that's true that RL resuls is very fluctuative, some times the score is decreasing but after few episodes it starts increasing again  
  > **username_3**: In Reinforcement Learning you do want to minimise/maximise a loss function!  
  > **username_1**: @RémyHosseinkhanBoucher not necessarily. Classic q learning for example require only a reward function, which is much different to a loss function conceptually.  
  
  ---
  
  ### Answer by username_2 on 2020-11-23. Score: 9
  
  [The accepted answer](https://ai.stackexchange.com/a/20135/2444) does not provide a good definition of *over-fitting*, which actually exists and is a defined concept in reinforcement learning too. For example, the paper [Quantifying Generalization in Reinforcement Learning](http://proceedings.mlr.press/v97/cobbe19a/cobbe19a.pdf) completely focuses on this issue. Let me give you more details.
  
  ### Over-fitting in supervised learning
  
  In **supervised learning (SL)**, **over-fitting** is defined as the difference (or gap) in the performance of the ML model (such as a neural network) on the training and test datasets. If the model performs significantly better on the training dataset than on the test dataset, then the ML model has over-fitted the training data. Consequently, it has not generalized (well enough) to other data other than the training data (i.e. the test data). The relationship between over-fitting and [generalization](https://ai.stackexchange.com/a/16737/2444) should now be clearer.
  
  ### Over-fitting in reinforcement learning
  
  In **reinforcement learning (RL)** (you can find a brief recap of what RL is [here](https://ai.stackexchange.com/a/20957/2444)), you want to find an **optimal policy** or value function (from which the policy can be derived), which can be represented by a neural network (or another model). A policy $\pi$ is optimal in environment $E$ if it leads to the highest cumulative reward in the long run in that environment $E$, which is often mathematically modelled as a (partially or fully observable) Markov decision process.
  
  In some cases, you are also interested in knowing whether your policy $\pi$ can also be used in a different environment than the environment it has been trained in, i.e. you're interested in knowing if the knowledge acquired in that training environment $E$ can be **transferred** to a different (but typically related) environment (or task) $E'$. For example, you may only be able to train your policy in a simulated environment (because of resource/safety constraints), then you want to transfer this learned policy to the real world. In those cases, you can define the concept of **over-fitting** in a similar way to the way we define over-fitting in SL. The only difference may be that you may say that the learned policy *has over-fitted the training environment* (rather than saying that the ML model has over-fitted the training dataset), but, given that the environment provides the *data*, then you could even say in RL that your policy has over-fitted the training data.
  
  ### Catastrophic forgetting
  
  There is also the issue of [**catastrophic forgetting (CF)**](https://ai.stackexchange.com/a/13293/2444) in RL, i.e., while learning, your RL agent may forget what it's previously learned, and this can even happen in the same environment. Why am I talking about CF? Because what it's happening to you is probably CF, i.e., *while learning*, the agent performs well for a while, then its performance drops (although I have read a paper that strangely defines CF differently in RL). You could also say that over-fitting is happening in your case, but, if you are continuously training and the performance changes, then CF is probably what you need to investigate. So, you should reserve the word *over-fitting* in RL when you're interested in **transfer learning** (i.e. the training and test environments do not coincide).
  
  
  ---
  
  