Answers.Body: <p>It is fairly common in DQN to train a minibatch after <em>every</em> observation received after the replay memory has enough data (how much is enough is yet another parameter). This is not necessary, and it is fine to collect more data between training steps, the alogrithm is still DQN. The value higher than 1 for <code>train_freq</code> here might be related to use of prioritised replay memory sampling - I have no real experience with that.</p>
  <p>The update to target network generally needs to occur less frequently than training steps, it is intended to stabilise results numerically, so that over or under estimates of value functions do not result in runaway feedback.</p>
  <p>The parameters choices <em>will</em> interact each other, most hyperparameters in machine learning do so unfortunately. Which makes searching for ideal values fiddly and time-consuming.</p>
  <p>In this case it is safe to say that <code>train_freq</code> is expected to be much lower than <code>target_update_freq</code>, probably by at least an order of magnitude, and more usually 2 or 3 orders of magnitude. However, that's not quite the same as saying there is a strong relationship between choices for those two hyperparameters. The value of <code>batch_size</code> is also relevant here, as it shows the rate that memory is being used (and re-used) by the training process.</p>
  <p>The library you are using has these defaults:</p>
  <pre><code>    batch_size::Int64 = 32
      train_freq::Int64 = 4
      target_update_freq::Int64 = 500
  </code></pre>
  <p>They seem like sane starting points. You are relatively free to change them as if they were independent, as there is no simple rule like &quot;target_update_freq should be 125 times train_freq&quot;. As a very rough guide, you can expect that high values of <code>train_freq</code>, low values of <code>batch_size</code> and low values of <code>target_update_freq</code> are likely to cause instability in the learning process, whilst going too far in the opposite direction may slow learning down. You might be able to set <code>train_freq</code> to 1, but I am not completely certain about that either in combination with the prioritised replay memory sampling which seems to be the default in the library you are using.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-18T14:56:16.307Z
Answers.LastActivityDate: 2020-06-18T16:25:42.043Z
Answers.LastEditDate: 2020-06-18T16:25:42.043Z
Answers.UserDisplayName: Neil Slater
Body: <p>In a Deep Q-learning algorithm, we perform a batch training every <code>train_freq</code> and we update the parameters of the target network every <code>target_update_freq</code>. Are <code>train_freq</code> and <code>target_update_freq</code> necessary related, e.g., one should be always greater than the other, or must they be independently optimized depending on the problem?</p>
  <p><strong>EDIT</strong> Changed the name of <code>batch_freq</code> to <code>train_freq</code>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T07:09:19.150Z
Comments.Text: It's more usual in my experience to run a moderately sized batch on every step. However, there is a lot of variation within the DQN method. Where are you seeing the `batch_freq` parameter? If you link the library or paper where this param is discussed I can probably figure out an answer
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T07:16:36.353Z
Comments.Text: If batch_freq is batch size then no. Neither in my experience nor in any paper I've read
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-18T14:03:14.280Z
Comments.Text: @NeilSlater I think it's the standard implementation of DQN. The code is from https://github.com/JuliaPOMDP/DeepQLearning.jl/blob/master/src/solver.jl. Line https://github.com/JuliaPOMDP/DeepQLearning.jl/blob/b590c378b622204c20c57e05a26cdd797a3cfedc/src/solver.jl#L132 is doing the minibatch training every `train_freq`. Then, line https://github.com/JuliaPOMDP/DeepQLearning.jl/blob/b590c378b622204c20c57e05a26cdd797a3cfedc/src/solver.jl#L138 is doing the update every `target_update_freq`.
Comments.UserDisplayName: zdm
Tags: <reinforcement-learning><deep-rl>
Title: In Deep Q-learning, are the target update frequency and the batch training frequency related?
UserDisplayName: zdm
ThreadText: # In Deep Q-learning, are the target update frequency and the batch training frequency related?
  
  (Asked by: username_0 on 2020-06-18)
  
  In a Deep Q-learning algorithm, we perform a batch training every `train_freq` and we update the parameters of the target network every `target_update_freq`. Are `train_freq` and `target_update_freq` necessary related, e.g., one should be always greater than the other, or must they be independently optimized depending on the problem?
  
  **EDIT** Changed the name of `batch_freq` to `train_freq`.
  
  **Tags:** reinforcement-learning, deep-rl
  > **Comments:**
  > **username_1**: It's more usual in my experience to run a moderately sized batch on every step. However, there is a lot of variation within the DQN method. Where are you seeing the `batch_freq` parameter? If you link the library or paper where this param is discussed I can probably figure out an answer  
  > **username_2**: If batch_freq is batch size then no. Neither in my experience nor in any paper I've read  
  > **username_0**: @NeilSlater I think it's the standard implementation of DQN. The code is from https://github.com/JuliaPOMDP/DeepQLearning.jl/blob/master/src/solver.jl. Line https://github.com/JuliaPOMDP/DeepQLearning.jl/blob/b590c378b622204c20c57e05a26cdd797a3cfedc/src/solver.jl#L132 is doing the minibatch training every `train_freq`. Then, line https://github.com/JuliaPOMDP/DeepQLearning.jl/blob/b590c378b622204c20c57e05a26cdd797a3cfedc/src/solver.jl#L138 is doing the update every `target_update_freq`.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-18. Score: 5 | Accepted answer
  
  It is fairly common in DQN to train a minibatch after every observation received after the replay memory has enough data (how much is enough is yet another parameter). This is not necessary, and it is fine to collect more data between training steps, the alogrithm is still DQN. The value higher than 1 for train_freq here might be related to use of prioritised replay memory sampling - I have no real experience with that.
  The update to target network generally needs to occur less frequently than training steps, it is intended to stabilise results numerically, so that over or under estimates of value functions do not result in runaway feedback.
  The parameters choices will interact each other, most hyperparameters in machine learning do so unfortunately. Which makes searching for ideal values fiddly and time-consuming.
  In this case it is safe to say that train_freq is expected to be much lower than target_update_freq, probably by at least an order of magnitude, and more usually 2 or 3 orders of magnitude. However, that's not quite the same as saying there is a strong relationship between choices for those two hyperparameters. The value of batch_size is also relevant here, as it shows the rate that memory is being used (and re-used) by the training process.
  The library you are using has these defaults:
      batch_size::Int64 = 32
      train_freq::Int64 = 4
      target_update_freq::Int64 = 500
  
  They seem like sane starting points. You are relatively free to change them as if they were independent, as there is no simple rule like "target_update_freq should be 125 times train_freq". As a very rough guide, you can expect that high values of train_freq, low values of batch_size and low values of target_update_freq are likely to cause instability in the learning process, whilst going too far in the opposite direction may slow learning down. You might be able to set train_freq to 1, but I am not completely certain about that either in combination with the prioritised replay memory sampling which seems to be the default in the library you are using.
  
  
  ---
  
  