Answers.Body: <p>You have a free choice to either:</p>
  <ul>
  <li><p>Wait until the replay buffer hits a minimum size for sampling.</p>
  </li>
  <li><p>Take smaller samples from the buffer initially, until the buffer is large enough. On the first time step the minibatch size will be 1.</p>
  </li>
  </ul>
  <p>Both these approaches are valid, and neither changes the DDPG algorithm significantly. Plenty of other hyperparameters for DDPG will have bigger effects on the end result.</p>
  <p>The Keras implementation of DDPG here <a href="https://keras.io/examples/rl/ddpg_pendulum/" rel="nofollow noreferrer">https://keras.io/examples/rl/ddpg_pendulum/</a> uses the second approach.</p>
  <p>The pseudocode in the paper is not 100% clear, but I would expect they did the same.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T13:38:04.153Z
Answers.LastActivityDate: 2022-07-03T13:43:51.953Z
Answers.LastEditDate: 2022-07-03T13:43:51.953Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am facing an issue in understanding the following line from the <a href="https://arxiv.org/pdf/1509.02971.pdf#page=5" rel="nofollow noreferrer">pseudocode of the DDPG algorithm</a></p>
  <blockquote>
  <p>Sample a random minibatch of <span class="math-container">$N$</span> transitions <span class="math-container">$(s_i, a_i, r_i, s_{i+1})$</span> from <span class="math-container">$R$</span></p>
  </blockquote>
  <p>Here <span class="math-container">$N$</span> is a hyperparameter that is equal to the number of transitions or samples that need to be present in the minibatch.</p>
  <p>Traditionally, we take a minibatch of samples from a dataset, which contains all samples, and pass them to a neural network, but if we observe the DDPG pseudocode, we are storing transition after transition into the buffer <span class="math-container">$R$</span>. So, I think, it needs several steps before sampling from the buffer <span class="math-container">$R$</span>. But, if we observe the pseudocode, it says that we need to sample from the first timestep of the first episode.</p>
  <p>How is it possible? Where am I missing?</p>
  
Tags: <deep-rl><implementation><ddpg><experience-replay><batch-learning>
Title: How to sample the tuples during the initial time steps of the DDPG algorithm?
UserDisplayName: hanugm
ThreadText: # How to sample the tuples during the initial time steps of the DDPG algorithm?
  
  (Asked by: username_0 on 2022-07-03)
  
  I am facing an issue in understanding the following line from the [pseudocode of the DDPG algorithm](https://arxiv.org/pdf/1509.02971.pdf#page=5)
  
  > Sample a random minibatch of $N$ transitions $(s\_i, a\_i, r\_i, s\_{i+1})$ from $R$
  
  Here $N$ is a hyperparameter that is equal to the number of transitions or samples that need to be present in the minibatch.
  
  Traditionally, we take a minibatch of samples from a dataset, which contains all samples, and pass them to a neural network, but if we observe the DDPG pseudocode, we are storing transition after transition into the buffer $R$. So, I think, it needs several steps before sampling from the buffer $R$. But, if we observe the pseudocode, it says that we need to sample from the first timestep of the first episode.
  
  How is it possible? Where am I missing?
  
  **Tags:** deep-rl, implementation, ddpg, experience-replay, batch-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-03. Score: 2 | Accepted answer
  
  You have a free choice to either:
  
  * Wait until the replay buffer hits a minimum size for sampling.
  * Take smaller samples from the buffer initially, until the buffer is large enough. On the first time step the minibatch size will be 1.
  
  Both these approaches are valid, and neither changes the DDPG algorithm significantly. Plenty of other hyperparameters for DDPG will have bigger effects on the end result.
  
  The Keras implementation of DDPG here <https://keras.io/examples/rl/ddpg_pendulum/> uses the second approach.
  
  The pseudocode in the paper is not 100% clear, but I would expect they did the same.
  
  
  ---
  
  