Answers.Body: <p>I have found the issue.</p>
  <p>Essentially, what appears to be happening is when I do my (only) 10 manual runs, I happen to get a bad sample of starting points.</p>
  <p>During training, for each episode, I randomly select a start point in the data so that the agent doesn't overfit to a single starting state. However, during my &quot;manual testing&quot;, I just chose 10 chronological starting points to run episodes from. It appears that the 10 chronological starting points just happen to be inconvenient/bad, so the results aren't great.</p>
  <p>To verify this idea, I ran 500 &quot;manual for-loop tests&quot; of the fully trained model by randomly selecting a starting point each time (just like I did during training). When I did this, the mean result across the 500 runs was quite similar to the final &quot;episode mean reward&quot; of my agent during training.</p>
  <p>So, it appears to come down to the &quot;luck of the draw&quot; when it comes to the episode starting points.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-01T15:52:32.810Z
Answers.LastActivityDate: 2022-07-01T15:52:32.810Z
Answers.UserDisplayName: Vladimir Belik
Body: <p>I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I would never expect an agent to get an episode score higher than 2.5 or 3).</p>
  <p>By the end of training (3 million time steps, ~1350 episodes), I can see that the agent has a &quot;mean episode reward&quot; of roughly 0.4 . For context, below is an image of the plotted of that mean episode reward over time.</p>
  <p><a href="https://i.sstatic.net/vTSNW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vTSNW.png" alt="enter image description here" /></a></p>
  <p>So, then, my expectation is that when I take this trained model and run it on an environment consisting of <em>the same data it was trained on</em>, it should have a performance somewhere around 0.4 reward on average per episode.</p>
  <p>However, this is not what I find. In my case, the fully trained agent's mean reward is -0.3 over 10 episodes (episode length equal to episode length during training) which collectively span the entirety of the training data.</p>
  <p>Why is the fully trained agent, who was just reported to have +0.4 mean score per episode, performing so poorly when run manually through a for-loop on the same training data?</p>
  <p>By &quot;manually through a for-loop&quot;, I mean that I have a for-loop through which I feed the agent observations one at a time, call <code>model.predict()</code> to get an action and then enact that action etc.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-29T23:22:46.867Z
Comments.Text: You are right, those two numbers should match. To me, it sounds like some unexpected behavior in the code, so I would carefully go through the code again and again. I don't think someone can identify what's wrong based on your description.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-30T00:25:06.883Z
Comments.Text: @Chillston I see. Thank you for your input. I'm pretty new to reinforcement learning, so at this stage, it's just hard for me to judge what's common/typical behavior vs. what's an error. I should note that my environment comes from a tabular time series dataset, and on every episode during training, I randomly choose a "starting point" from the dataset. In "testing", I naturally do not choose the starting points randomly, but chronologically. So, maybe this randomness is the cause of the issue (maybe I "happen" to choose "bad" start points)?
Comments.UserDisplayName: Vladimir Belik
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-30T00:27:27.557Z
Comments.Text: @Chillston Perhaps I will run a few thousand episodes in "test" mode, randomly selecting a start point each time, to see if there's a difference between the average "test" result vs. the mean episode reward during training (to negate the effect of potentially unfortunate choice of starting points). Should I add some of my code to the question?
Comments.UserDisplayName: Vladimir Belik
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-30T20:54:55.983Z
Comments.Text: @Chillston It appears that that's the issue - when I run a bunch of trials with random starting points, the mean "manual" score is about the same as the mean episode reward.
Comments.UserDisplayName: Vladimir Belik
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-01T17:44:06.243Z
Comments.Text: Very nice, that makes a lot of sense! :)
  Regarding the addition of code - in this case the question is more about debugging, this you should post on StackOverflow. But you sorted it out already anyways.
Comments.UserDisplayName: Chillston
Tags: <comparison><deep-rl><proximal-policy-optimization><performance>
Title: Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?
UserDisplayName: Vladimir Belik
ThreadText: # Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?
  
  (Asked by: username_0 on 2022-06-27)
  
  I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I would never expect an agent to get an episode score higher than 2.5 or 3).
  
  By the end of training (3 million time steps, ~1350 episodes), I can see that the agent has a "mean episode reward" of roughly 0.4 . For context, below is an image of the plotted of that mean episode reward over time.
  
  [![enter image description here](https://i.sstatic.net/vTSNW.png)](https://i.sstatic.net/vTSNW.png)
  
  So, then, my expectation is that when I take this trained model and run it on an environment consisting of *the same data it was trained on*, it should have a performance somewhere around 0.4 reward on average per episode.
  
  However, this is not what I find. In my case, the fully trained agent's mean reward is -0.3 over 10 episodes (episode length equal to episode length during training) which collectively span the entirety of the training data.
  
  Why is the fully trained agent, who was just reported to have +0.4 mean score per episode, performing so poorly when run manually through a for-loop on the same training data?
  
  By "manually through a for-loop", I mean that I have a for-loop through which I feed the agent observations one at a time, call `model.predict()` to get an action and then enact that action etc.
  
  **Tags:** comparison, deep-rl, proximal-policy-optimization, performance
  
  > **Comments:**
  > **username_1**: You are right, those two numbers should match. To me, it sounds like some unexpected behavior in the code, so I would carefully go through the code again and again. I don't think someone can identify what's wrong based on your description.  
  > **username_0**: @username_1 I see. Thank you for your input. I'm pretty new to reinforcement learning, so at this stage, it's just hard for me to judge what's common/typical behavior vs. what's an error. I should note that my environment comes from a tabular time series dataset, and on every episode during training, I randomly choose a "starting point" from the dataset. In "testing", I naturally do not choose the starting points randomly, but chronologically. So, maybe this randomness is the cause of the issue (maybe I "happen" to choose "bad" start points)?  
  > **username_0**: @username_1 Perhaps I will run a few thousand episodes in "test" mode, randomly selecting a start point each time, to see if there's a difference between the average "test" result vs. the mean episode reward during training (to negate the effect of potentially unfortunate choice of starting points). Should I add some of my code to the question?  
  > **username_0**: @username_1 It appears that that's the issue - when I run a bunch of trials with random starting points, the mean "manual" score is about the same as the mean episode reward.  
  > **username_1**: Very nice, that makes a lot of sense! :) Regarding the addition of code - in this case the question is more about debugging, this you should post on StackOverflow. But you sorted it out already anyways.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-07-01. Score: 1 | Accepted answer
  
  I have found the issue.
  
  Essentially, what appears to be happening is when I do my (only) 10 manual runs, I happen to get a bad sample of starting points.
  
  During training, for each episode, I randomly select a start point in the data so that the agent doesn't overfit to a single starting state. However, during my "manual testing", I just chose 10 chronological starting points to run episodes from. It appears that the 10 chronological starting points just happen to be inconvenient/bad, so the results aren't great.
  
  To verify this idea, I ran 500 "manual for-loop tests" of the fully trained model by randomly selecting a starting point each time (just like I did during training). When I did this, the mean result across the 500 runs was quite similar to the final "episode mean reward" of my agent during training.
  
  So, it appears to come down to the "luck of the draw" when it comes to the episode starting points.
  
  
  ---
  
  