Answers.Body: <p>Potentially.</p>
  
  <p>If you do offline reinforcement learning, you're basically learning to approximate a function by sampling input/output pairs, rather than episode-by-episode. Here, your batch size could be set exactly as in an ordinary supervised learning problem.</p>
  
  <p>If you do online learning, then it's not clear to me that the techniques used to set the learning rate in supervised learning can be directly applied though.</p>
  
  <p>Both approaches are well covered in the RL chapter of <a href="https://rads.stackoverflow.com/amzn/click/0136042597" rel="nofollow noreferrer">Russell &amp; Norvig</a> (17? 18?).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-08T00:11:04.443Z
Answers.LastActivityDate: 2018-08-08T00:11:04.443Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>From my understanding of reinforcement learning, you will have an agent and an environment.</p>
  <p>In each episode, the agent observes the state <span class="math-container">$s$</span>, takes some action action <span class="math-container">$a$</span>, then gets some reward <span class="math-container">$r$</span>, and finally observes the next state <span class="math-container">$s'$</span>, and do it again and again until the end of the episode.</p>
  <p>The above process does not incur any &quot;learning&quot;. Then when and where exactly do you &quot;learn&quot;? You learn from your history. In traditional Q learning, the Q matrix is updated every time you have a new observation of <span class="math-container">$(s_t, a_t, r_t, s'_{t+1})$</span>. Just like the supervised learning, you put in training sample one by one.</p>
  <p>Similarly, you can feed in training samples in &quot;batch&quot; when you train, which means you &quot;remember&quot; the past <span class="math-container">$N$</span> observations and train them together. I think that is the answer to your question.</p>
  <p>Furthermore, the past <span class="math-container">$N$</span> observations could have a strong correlation that you don't want. To break this, you may have a larger &quot;memory&quot; that stores many observations, and you only sample a few (this number is your new batch size) randomly every time you train your model. This is called <em>experience replay</em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-08T01:13:02.393Z
Answers.LastActivityDate: 2020-06-19T16:41:48.613Z
Answers.LastEditDate: 2020-06-19T16:41:48.613Z
Answers.UserDisplayName: Kevin. Fang
Body: <p>I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up on methods of programmatically choosing a learning rate. I came across this blog post, <a href="https://medium.com/@nachiket.tanksale/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6" rel="nofollow noreferrer"><em>Finding Good Learning Rate and The One Cycle Policy</em></a>, about finding cyclical learning rate and finding good bounds for learning rates.</p>
  <p>All the articles about this method talk about measuring loss across batches in the data. However, as I understand it, in <a href="https://en.wikipedia.org/wiki/Reinforcement_learning" rel="nofollow noreferrer">Reinforcement Learning</a> tasks do not really have any &quot;batches&quot;, they just have episodes that can be generated by an environment as many times as one wants, which also gives rewards that are then used to optimize the network.</p>
  <p>Is there a way to translate the concept of batch size into reinforcement learning, or a way to use this method of cyclical learning rates with reinforcement learning?</p>
  
Tags: <neural-networks><deep-learning><reinforcement-learning><learning-rate><batch-size>
Title: Is there a way to translate the concept of batch size into reinforcement learning?
UserDisplayName: user10011538
ThreadText: # Is there a way to translate the concept of batch size into reinforcement learning?
  
  (Asked by: username_0 on 2018-08-07)
  
  I am using a neural network as my function approximator for reinforcement learning. In order to get it to train well, I need to choose a good learning rate. Hand-picking one is difficult, so I read up on methods of programmatically choosing a learning rate. I came across this blog post, [*Finding Good Learning Rate and The One Cycle Policy*](https://medium.com/@nachiket.tanksale/finding-good-learning-rate-and-the-one-cycle-policy-7159fe1db5d6), about finding cyclical learning rate and finding good bounds for learning rates.
  
  All the articles about this method talk about measuring loss across batches in the data. However, as I understand it, in [Reinforcement Learning](https://en.wikipedia.org/wiki/Reinforcement_learning) tasks do not really have any "batches", they just have episodes that can be generated by an environment as many times as one wants, which also gives rewards that are then used to optimize the network.
  
  Is there a way to translate the concept of batch size into reinforcement learning, or a way to use this method of cyclical learning rates with reinforcement learning?
  
  **Tags:** neural-networks, deep-learning, reinforcement-learning, learning-rate, batch-size
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-08-08. Score: 3
  
  Potentially.
  
  If you do offline reinforcement learning, you're basically learning to approximate a function by sampling input/output pairs, rather than episode-by-episode. Here, your batch size could be set exactly as in an ordinary supervised learning problem.
  
  If you do online learning, then it's not clear to me that the techniques used to set the learning rate in supervised learning can be directly applied though.
  
  Both approaches are well covered in the RL chapter of [Russell & Norvig](https://rads.stackoverflow.com/amzn/click/0136042597) (17? 18?).
  
  
  ---
  
  ### Answer by username_2 on 2018-08-08. Score: 2
  
  From my understanding of reinforcement learning, you will have an agent and an environment.
  
  In each episode, the agent observes the state $s$, takes some action action $a$, then gets some reward $r$, and finally observes the next state $s'$, and do it again and again until the end of the episode.
  
  The above process does not incur any "learning". Then when and where exactly do you "learn"? You learn from your history. In traditional Q learning, the Q matrix is updated every time you have a new observation of $(s\_t, a\_t, r\_t, s'\_{t+1})$. Just like the supervised learning, you put in training sample one by one.
  
  Similarly, you can feed in training samples in "batch" when you train, which means you "remember" the past $N$ observations and train them together. I think that is the answer to your question.
  
  Furthermore, the past $N$ observations could have a strong correlation that you don't want. To break this, you may have a larger "memory" that stores many observations, and you only sample a few (this number is your new batch size) randomly every time you train your model. This is called *experience replay*.
  
  
  ---
  
  