Answers.Body: <p>Episodes are discrete, there is no need for calculus. Your &quot;sample efficiency&quot; metric is:</p>
  <p><span class="math-container">$$\sum_{x=a}^b R_x$$</span></p>
  <p>The quantity you are measuring per episode is the return (undiscounted). The sum of this over many episodes does not measure sample efficiency as the term is usually meant, although the sample efficiency of the algorithm you use should impact the numbers you see. Getting a high value of this metric, averaged over many training runs, implies two things:</p>
  <ul>
  <li><p>The algorithm learns to exploit the environment quickly. This is related to sample efficiency.</p>
  </li>
  <li><p>The algorithm does not pay a high cost for exploring. This is not directly related to sample efficiency, and may in fact be in conflict with learning an environment quickly.</p>
  </li>
  </ul>
  <p>These are both desirable properties of a reinforcement learning algorithm. They often need to be considered in balance, this is the <a href="https://www.manifold.ai/exploration-vs-exploitation-in-reinforcement-learning" rel="nofollow noreferrer">exploration versus exploitation dilemma</a> in RL, which can be studied in a simplified form in mult-arm bandit problems. You may be able to take inspiration from how bandit algorithms are measured for other metrics related to efficient learning.</p>
  <p>Your metric is most useful when considering learning agents run in a live environment, where costs of mistakes during learning are real.</p>
  <p>If instead you are training an agent in a safe environment - e.g. in simulation - then you may not be interested in the undiscounted returns <span class="math-container">$R_x$</span> during training. Your goal may be to train the agent using the least CPU time, the least number of simulation runs etc. In which case, you care most about the mean return achievable by the trained algorithm after spending a certain amount of whatever resource you are managing. This is more closely related to the concept of sample efficiency, and to measure that you could plot the returns from separate <em>test</em> episodes at routine intervals, with exploration removed (e.g. if you were using DQN, then with <span class="math-container">$\epsilon = 0$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-28T16:45:00.350Z
Answers.Text: Thanks for this. I am using policy gradient algorithms like SAC and TD3 for continuous action space. Episode has NO terminal state so all episodes are of the same length. Which sample efficiency metric do you think I'd use? And yes, I am using a simulation
Answers.UserDisplayName: bluewander
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-28T19:24:21.463Z
Answers.Text: I would use a metric of minimising the resource spend that I am most concerned about, whilst reaching either acceptable, near optimal or optimal behaviour, averaged over enough runs that differences between approaches can be shown to be significant. So if you are truly interested only in number of simulated time steps required (a fairly pure measure of sample efficiency), then count the number of simulated time steps required to reach some close-to-optimal mean reward per time step.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-28T06:59:51.923Z
Answers.LastActivityDate: 2021-06-28T07:21:22.627Z
Answers.LastEditDate: 2021-06-28T07:21:22.627Z
Answers.UserDisplayName: Neil Slater
Body: <p>I define sample efficiency as <strong>the area under the curve/graph</strong>, where <span class="math-container">$x$</span>-axis is the number of episodes while y-axis is the cumulative reward for that episode. <em><strong>I would like to formally define it with a mathematical function.</strong></em></p>
  <p>If the notation for cumulative reward for <span class="math-container">$x$</span>th episode is:</p>
  <p><span class="math-container">$$R_x = \sum_{t=0}^{t=T} r_t,$$</span></p>
  <p>where <span class="math-container">$r_t$</span> is the reward for timestep <span class="math-container">$t$</span> and <span class="math-container">$T$</span> is the max number if steps per episode.</p>
  <p>So is the equation for area under the graph/curve the one below?</p>
  <p><span class="math-container">$$\text{Sample Efficiency} =\int_{a}^{b} R_x \ dx$$</span></p>
  <p>I will be just using a Python library to get the area under the graph which uses Simpson's rule for integrating.</p>
  
Tags: <reinforcement-learning><deep-rl><rewards><return><sample-efficiency>
Title: How do I represent sample efficiency of RL rewards in mathematical notation?
UserDisplayName: bluewander
ThreadText: # How do I represent sample efficiency of RL rewards in mathematical notation?
  
  (Asked by: username_0 on 2021-06-28)
  
  I define sample efficiency as **the area under the curve/graph**, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. ***I would like to formally define it with a mathematical function.***
  
  If the notation for cumulative reward for $x$th episode is:
  
  $$R\_x = \sum\_{t=0}^{t=T} r\_t,$$
  
  where $r\_t$ is the reward for timestep $t$ and $T$ is the max number if steps per episode.
  
  So is the equation for area under the graph/curve the one below?
  
  $$\text{Sample Efficiency} =\int\_{a}^{b} R\_x \ dx$$
  
  I will be just using a Python library to get the area under the graph which uses Simpson's rule for integrating.
  
  **Tags:** reinforcement-learning, deep-rl, rewards, return, sample-efficiency
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-06-28. Score: 1
  
  Episodes are discrete, there is no need for calculus. Your "sample efficiency" metric is:
  
  $$\sum\_{x=a}^b R\_x$$
  
  The quantity you are measuring per episode is the return (undiscounted). The sum of this over many episodes does not measure sample efficiency as the term is usually meant, although the sample efficiency of the algorithm you use should impact the numbers you see. Getting a high value of this metric, averaged over many training runs, implies two things:
  
  * The algorithm learns to exploit the environment quickly. This is related to sample efficiency.
  * The algorithm does not pay a high cost for exploring. This is not directly related to sample efficiency, and may in fact be in conflict with learning an environment quickly.
  
  These are both desirable properties of a reinforcement learning algorithm. They often need to be considered in balance, this is the [exploration versus exploitation dilemma](https://www.manifold.ai/exploration-vs-exploitation-in-reinforcement-learning) in RL, which can be studied in a simplified form in mult-arm bandit problems. You may be able to take inspiration from how bandit algorithms are measured for other metrics related to efficient learning.
  
  Your metric is most useful when considering learning agents run in a live environment, where costs of mistakes during learning are real.
  
  If instead you are training an agent in a safe environment - e.g. in simulation - then you may not be interested in the undiscounted returns $R\_x$ during training. Your goal may be to train the agent using the least CPU time, the least number of simulation runs etc. In which case, you care most about the mean return achievable by the trained algorithm after spending a certain amount of whatever resource you are managing. This is more closely related to the concept of sample efficiency, and to measure that you could plot the returns from separate *test* episodes at routine intervals, with exploration removed (e.g. if you were using DQN, then with $\epsilon = 0$
  
  > **Comments:**
  > **username_0**: Thanks for this. I am using policy gradient algorithms like SAC and TD3 for continuous action space. Episode has NO terminal state so all episodes are of the same length. Which sample efficiency metric do you think I'd use? And yes, I am using a simulation  
  > **username_1**: I would use a metric of minimising the resource spend that I am most concerned about, whilst reaching either acceptable, near optimal or optimal behaviour, averaged over enough runs that differences between approaches can be shown to be significant. So if you are truly interested only in number of simulated time steps required (a fairly pure measure of sample efficiency), then count the number of simulated time steps required to reach some close-to-optimal mean reward per time step.  
  
  ---
  
  