Answers.Body: <p>You have not linked the paper where you got such images, however, usually in DRL publications with these environments, the x-axis refers to training-steps, not environment-steps</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-28T13:18:46.653Z
Answers.Text: Here is the paper: https://arxiv.org/abs/2010.02966
Answers.UserDisplayName: waylone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-06T06:24:20.077Z
Answers.Text: I'm new to DRL. If x axis refers to training steps,then how to calculate return at each training step?Should I use the policy learned at this training step to interact with the environment for a complete episode to get return of current training step? Thank you.
Answers.UserDisplayName: waylone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-27T11:02:49.850Z
Answers.LastActivityDate: 2023-09-27T11:02:49.850Z
Answers.UserDisplayName: Alberto
Body: <p>I know return is total discounted reward of an episode,so it is easy to plot return vs. episode.But in many papers,they provide figures of average return vs step,like this:<a href="https://i.sstatic.net/vuZul.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vuZul.png" alt="enter image description here" /></a></p>
  <p>Based on my knowledge,return is discounted cumulative rewards of an episode,it can't be calculated until the end of episodes,so I feel puzzled about how to calculate average return at each training step.</p>
  <p>Does return of each training step mean discounted cumulative rewards collected from current training step to the end of current episode?If an agent is trained with N episodes,and each episode has T step,then I can calculate return of the (i-1)<em>Tth step to the i</em>Tth step when the ith episode ends and plot return vs. step if all N episodes finish,is it right?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-27T11:24:05.100Z
Comments.Text: Please link the paper where you found the graphs. Answers can then explain those graphs with more confidence. The answers may also apply to other graphs you have seen, but you should always check the paper details  to find what definitions are being used
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-28T13:19:54.453Z
Comments.Text: Here is the paper: https://arxiv.org/abs/2010.02966
Comments.UserDisplayName: waylone
Tags: <reinforcement-learning>
Title: How to plot average return vs step figures in reinforcement learning？
UserDisplayName: waylone
ThreadText: # How to plot average return vs step figures in reinforcement learning？
  
  (Asked by: username_0 on 2023-09-27)
  
  I know return is total discounted reward of an episode,so it is easy to plot return vs. episode.But in many papers,they provide figures of average return vs step,like this:[![enter image description here](https://i.sstatic.net/vuZul.png)](https://i.sstatic.net/vuZul.png)
  
  Based on my knowledge,return is discounted cumulative rewards of an episode,it can't be calculated until the end of episodes,so I feel puzzled about how to calculate average return at each training step.
  
  Does return of each training step mean discounted cumulative rewards collected from current training step to the end of current episode?If an agent is trained with N episodes,and each episode has T step,then I can calculate return of the (i-1)*Tth step to the i*Tth step when the ith episode ends and plot return vs. step if all N episodes finish,is it right?
  
  **Tags:** reinforcement-learning
  > **Comments:**
  > **username_1**: Please link the paper where you found the graphs. Answers can then explain those graphs with more confidence. The answers may also apply to other graphs you have seen, but you should always check the paper details  to find what definitions are being used  
  > **username_0**: Here is the paper: https://arxiv.org/abs/2010.02966  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-09-27. Score: 1
  
  You have not linked the paper where you got such images, however, usually in DRL publications with these environments, the x-axis refers to training-steps, not environment-steps
  
  > **Comments:**
  > **username_0**: Here is the paper: https://arxiv.org/abs/2010.02966  
  > **username_0**: I'm new to DRL. If x axis refers to training steps,then how to calculate return at each training step?Should I use the policy learned at this training step to interact with the environment for a complete episode to get return of current training step? Thank you.  
  
  ---
  
  