Answers.Body: <p>Pythons <code>random.choice</code> may return the same sample within a single batch. Although you could view this as redundant data, it is still a fair sample statistically, so does not adversely affect results. In addition, the agent may already have sampled the same thing multiple times, resulting in repeated entries in memory. For the best <em>approximation</em>, this is often desirable - approximation quality is usually considered in terms of loss weighted by frequency of data (or sample density with continuous inputs).*</p>
  <p>You could easily code a sample without replacement approach if you want - the difference in performance will be undetectable. So most people implementing this don't worry about this detail.</p>
  <hr />
  <p>* In RL, there is a dilemma closely related to the exploration vs exploitation problem. Training a value approximator only on a target optimal policy's state/action distribution will produce the best approximations for action values for the real state distribution and on-policy decisions. However, it will tend to forget off-policy details, and may perform much worse as a result (even though it would have a low cost metric if treated as a supervised learning problem on predicting values for the optimal policy).</p>
  <p>Hence training with at least a little off-policy data, compromising accuracy for resilience, is often preferred. Taking it further are approaches like prioritised sampling which ignore experienced distribution of states anyway. So using &quot;fair&quot; samples is not some gold standard in RL, often we deliberately choose alternatives.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-25T07:38:02.917Z
Answers.LastActivityDate: 2023-08-25T10:08:50.007Z
Answers.LastEditDate: 2023-08-25T10:08:50.007Z
Answers.UserDisplayName: Neil Slater
Body: <p>When implementing batch sampling in RL learning, I saw that we are sampling with repetition, we use <code>np.random.choice(record_range, batch_size)</code> in Python for example, that can return the same element multiple times. Why one would sample the element more than once? Isn't this useless?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-26T01:50:38.910Z
Comments.Text: Example: let's say you have this distribution: 10 As, 1 billion Bs, 1 C. You sample without replacement 3 times. What distribution do you have? Is that representative of the actual one?
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><batch-size>
Title: Isn't it useless to batch with repetition in RL?
UserDisplayName: zdm
ThreadText: # Isn't it useless to batch with repetition in RL?
  
  (Asked by: username_0 on 2023-08-25)
  
  When implementing batch sampling in RL learning, I saw that we are sampling with repetition, we use `np.random.choice(record_range, batch_size)` in Python for example, that can return the same element multiple times. Why one would sample the element more than once? Isn't this useless?
  
  **Tags:** reinforcement-learning, batch-size
  
  > **Comments:**
  > **username_1**: Example: let's say you have this distribution: 10 As, 1 billion Bs, 1 C. You sample without replacement 3 times. What distribution do you have? Is that representative of the actual one?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-25. Score: 5 | Accepted answer
  
  Pythons `random.choice` may return the same sample within a single batch. Although you could view this as redundant data, it is still a fair sample statistically, so does not adversely affect results. In addition, the agent may already have sampled the same thing multiple times, resulting in repeated entries in memory. For the best *approximation*, this is often desirable - approximation quality is usually considered in terms of loss weighted by frequency of data (or sample density with continuous inputs).\*
  
  You could easily code a sample without replacement approach if you want - the difference in performance will be undetectable. So most people implementing this don't worry about this detail.
  
  ---
  
  \* In RL, there is a dilemma closely related to the exploration vs exploitation problem. Training a value approximator only on a target optimal policy's state/action distribution will produce the best approximations for action values for the real state distribution and on-policy decisions. However, it will tend to forget off-policy details, and may perform much worse as a result (even though it would have a low cost metric if treated as a supervised learning problem on predicting values for the optimal policy).
  
  Hence training with at least a little off-policy data, compromising accuracy for resilience, is often preferred. Taking it further are approaches like prioritised sampling which ignore experienced distribution of states anyway. So using "fair" samples is not some gold standard in RL, often we deliberately choose alternatives.
  
  
  ---
  
  