Answers.Body: <p>First of all, let's recall some definitions.</p>
  <p>A <em>sample</em> in the context of (deep) RL is a tuple <span class="math-container">$(s_t, a_t, r_t, s_{t+1})$</span> representing the information associated with a single interaction with the environment.</p>
  <p>As for <em>sample efficiency</em>, it is defined as follows <a href="https://www.sciencedirect.com/science/article/pii/S1364661319300610" rel="nofollow noreferrer">[1]</a>:</p>
  <blockquote>
  <p>Sample efficiency refers to the amount of data required for a learning system to attain any chosen target level of performance.</p>
  </blockquote>
  <p>So, how you measure it is closely related to the way it is defined.</p>
  <p>For example, one way to do it would be as shown in the figure below:</p>
  <ul>
  <li><p>On the y-axis you have the performance of your RL algorithm (e.g. in terms of average return over episodes as done in <a href="https://arxiv.org/pdf/1611.02247.pdf" rel="nofollow noreferrer">[2]</a>, or mean total episode reward across different environment runs as done in <a href="https://papers.nips.cc/paper/8044-sample-efficient-reinforcement-learning-with-stochastic-ensemble-value-expansion.pdf" rel="nofollow noreferrer">[3]</a>)</p>
  </li>
  <li><p>On the x-axis you have the number of samples that you took.</p>
  </li>
  <li><p>The dashed line corresponds to your performance baseline (e.g. the performance at which a certain game or any other RL environment is considered solved).</p>
  </li>
  </ul>
  <p>So, you can measure sample efficiency at the intersection, where you get the number of samples needed to reach the performance baseline. So, an algorithm that requires less samples would be more sample-efficient.</p>
  <p><a href="https://i.sstatic.net/Rqce9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Rqce9.png" alt="enter image description here" /></a></p>
  <p>Another way to do it would be the other way around, i.e. the RL agent is provided with a limited <em>budget</em> for the number of samples it could take. As a result, you can measure sample efficiency my measuring the area under the curve, as illustrated below. So, that would be how much performance you got just by using those samples in the budget. An algorithm that achieves a higher performance than another with the same amount of samples would then be more sample efficient.</p>
  <p><a href="https://i.sstatic.net/3MIDR.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3MIDR.png" alt="enter image description here" /></a></p>
  <p>I am not aware if there exist RL libraries that would provide you with this measure out-of-the-box. However, if you're using Python for example, I believe that using libraries like scipy or scikit-learn along with matplotlib could do the job.</p>
  <p>NB: Image credits go to the following presentation: <a href="https://www.youtube.com/watch?v=DT2MbIeec4U" rel="nofollow noreferrer">DLRLSS 2019 - Sample Efficient RL - Harm Van Seijen</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-18T12:54:32.147Z
Answers.LastActivityDate: 2020-06-18T15:25:59.703Z
Answers.LastEditDate: 2020-06-18T15:25:59.703Z
Answers.UserDisplayName: user5093249
Body: <p>I want to know if there is any metric to use for measuring sample-efficiency of a reinforcement learning algorithm? From reading research papers, I see claims that proposed models are more sample efficient but how does one reach this conclusion when comparing reinforcement learning algorithms?</p>
  
Tags: <reinforcement-learning><sample-efficiency>
Title: How to measure sample efficiency of a reinforcement learning algorithm?
UserDisplayName: rert588
ThreadText: # How to measure sample efficiency of a reinforcement learning algorithm?
  
  (Asked by: username_0 on 2020-06-18)
  
  I want to know if there is any metric to use for measuring sample-efficiency of a reinforcement learning algorithm? From reading research papers, I see claims that proposed models are more sample efficient but how does one reach this conclusion when comparing reinforcement learning algorithms?
  
  **Tags:** reinforcement-learning, sample-efficiency
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-18. Score: 5 | Accepted answer
  
  First of all, let's recall some definitions.
  
  A *sample* in the context of (deep) RL is a tuple $(s\_t, a\_t, r\_t, s\_{t+1})$ representing the information associated with a single interaction with the environment.
  
  As for *sample efficiency*, it is defined as follows [[1]](https://www.sciencedirect.com/science/article/pii/S1364661319300610):
  
  > Sample efficiency refers to the amount of data required for a learning system to attain any chosen target level of performance.
  
  So, how you measure it is closely related to the way it is defined.
  
  For example, one way to do it would be as shown in the figure below:
  
  * On the y-axis you have the performance of your RL algorithm (e.g. in terms of average return over episodes as done in [[2]](https://arxiv.org/pdf/1611.02247.pdf), or mean total episode reward across different environment runs as done in [[3]](https://papers.nips.cc/paper/8044-sample-efficient-reinforcement-learning-with-stochastic-ensemble-value-expansion.pdf))
  * On the x-axis you have the number of samples that you took.
  * The dashed line corresponds to your performance baseline (e.g. the performance at which a certain game or any other RL environment is considered solved).
  
  So, you can measure sample efficiency at the intersection, where you get the number of samples needed to reach the performance baseline. So, an algorithm that requires less samples would be more sample-efficient.
  
  [![enter image description here](https://i.sstatic.net/Rqce9.png)](https://i.sstatic.net/Rqce9.png)
  
  Another way to do it would be the other way around, i.e. the RL agent is provided with a limited *budget* for the number of samples it could take. As a result, you can measure sample efficiency my measuring the area under the curve, as illustrated below. So, that would be how much performance you got just by using those samples in the budget. An algorithm that achieves a higher performance than another with the same amount of samples would then be more sample efficient.
  
  [![enter image description here](https://i.sstatic.net/3MIDR.png)](https://i.sstatic.net/3MIDR.png)
  
  I am not aware if there exist RL libraries that would provide you with this measure out-of-the-box. However, if you're using Python for example, I believe that using libraries like scipy or scikit-learn along with matplotlib could do the job.
  
  NB: Image credits go to the following presentation: [DLRLSS 2019 - Sample Efficient RL - Harm Van Seijen](https://www.youtube.com/watch?v=DT2MbIeec4U)
  
  
  ---
  
  