Answers.Body: <p>I assume this is because you don't want to modify your training environment while evaluating. For example, if your evaluation triggers halfway through a training episode, the evaluation will need to call things like env.reset(), which will mess up the state of your training environment. It might be possible to cache the state of your training environment before evaluation, but this is much more complicated than just having separate training and evaluation environments.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-30T07:39:26.513Z
Answers.LastActivityDate: 2023-08-30T07:39:26.513Z
Answers.UserDisplayName: Andrei Muresanu
Body: <p>I am training an agent (stable baselines3 algorithm) on a custom environment. During training, I want to have a callback so that for every <span class="math-container">$N$</span> steps of the learning process, I get the current model and run it on my environment <span class="math-container">$M$</span> times and log the average reward received, with sb3 <a href="https://stable-baselines3.readthedocs.io/en/master/common/evaluation.html" rel="nofollow noreferrer">evaluate_policy</a> with deterministic set to true.</p>
  <p>I have read that you should run the agent on a separate validation environment, not the same instance of the environment that the agent trains on, e.g. here <a href="https://stable-baselines3.readthedocs.io/en/v1.0/guide/rl_tips.html#how-to-evaluate-an-rl-algorithm" rel="nofollow noreferrer">https://stable-baselines3.readthedocs.io/en/v1.0/guide/rl_tips.html#how-to-evaluate-an-rl-algorithm</a></p>
  <blockquote>
  <p>Because most algorithms use exploration noise during training, you need a separate test environment to evaluate the performance of your agent at a given time.</p>
  </blockquote>
  <p>I don't understand the reasoning for this. I don't know why exploration noise affects what environment I should use and I can't figure out why its not possible to use the same instance of the environment class when evaluating the model. There is no variable in my environment that is specific to training / testing.</p>
  <p>Is evaluating on the same environment instance as the model is trained on fine and I'm misinterpreting the general advice, or will my conclusions be wrong if I do it?</p>
  <p>The reason I am asking is because I can only have one environment instance running at any one time.</p>
  
Tags: <reinforcement-learning><deep-rl><intelligent-agent><environment><stable-baselines>
Title: Why is it recommended to use a "separate test environment" when evaluating a model?
UserDisplayName: jgklsdjfgkldsfaSDF
ThreadText: # Why is it recommended to use a "separate test environment" when evaluating a model?
  
  (Asked by: username_0 on 2022-12-09)
  
  I am training an agent (stable baselines3 algorithm) on a custom environment. During training, I want to have a callback so that for every $N$ steps of the learning process, I get the current model and run it on my environment $M$ times and log the average reward received, with sb3 [evaluate\_policy](https://stable-baselines3.readthedocs.io/en/master/common/evaluation.html) with deterministic set to true.
  
  I have read that you should run the agent on a separate validation environment, not the same instance of the environment that the agent trains on, e.g. here <https://stable-baselines3.readthedocs.io/en/v1.0/guide/rl_tips.html#how-to-evaluate-an-rl-algorithm>
  
  > Because most algorithms use exploration noise during training, you need a separate test environment to evaluate the performance of your agent at a given time.
  
  I don't understand the reasoning for this. I don't know why exploration noise affects what environment I should use and I can't figure out why its not possible to use the same instance of the environment class when evaluating the model. There is no variable in my environment that is specific to training / testing.
  
  Is evaluating on the same environment instance as the model is trained on fine and I'm misinterpreting the general advice, or will my conclusions be wrong if I do it?
  
  The reason I am asking is because I can only have one environment instance running at any one time.
  
  **Tags:** reinforcement-learning, deep-rl, intelligent-agent, environment, stable-baselines
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-30. Score: 0
  
  I assume this is because you don't want to modify your training environment while evaluating. For example, if your evaluation triggers halfway through a training episode, the evaluation will need to call things like env.reset(), which will mess up the state of your training environment. It might be possible to cache the state of your training environment before evaluation, but this is much more complicated than just having separate training and evaluation environments.
  
  
  ---
  
  