Answers.Body: <p>Most of papers and my experience support hard update, once per N steps. N is usually very bug, range in between 10^4 to 10^6. DQN training is slow. But that depend on the problem. If you DQN converge with soft update with weight ~.1 (N~10) your problem could be very simple.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-01T04:02:23.200Z
Answers.Text: Would you mind put some more detailed research of yours? I would like to see something like this technique has been used in 2019 by the SOTA or so. Thanks.
Answers.UserDisplayName: Phizaz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T06:36:23.803Z
Answers.Text: I was doing DQN for car driving simulation. The main points: even thogh DQN is off policy it's effectively act similar to on-policy due to overfitting on latest samples(latest measired in millions). That could be helped by increasing target net update intervaland to some degree size of replay buffer. Its very important to randomly remix samples from replay buffer then assembling training minibatch. Overfitting causing DQN drift, wich is main enemy
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-28T12:54:08.330Z
Answers.LastActivityDate: 2019-02-28T12:54:08.330Z
Answers.UserDisplayName: mirror2image
Body: <p>The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with <em>semi-gradient</em> objectives. In Deep Q network, semi-gradient TD is used and with experience replay the training could diverge. </p>
  
  <p>Target network is a slow changing network designed to slowly track the main value network. In <a href="https://arxiv.org/abs/1312.5602" rel="nofollow noreferrer">Mnih 2013</a>, it was designed to match the main network every <span class="math-container">$N$</span> steps. There is another way which slowly updates the weight in the direction to match the main network every step. To someone, the latter is called <em>Polyak updates</em>. </p>
  
  <p>I have done some very limited experiments and seen that with the same update rate, e.g. <span class="math-container">$N=10$</span>, Polyak update would update with the rate of 0.1, I usually see Polyak updates to give <em>smoother</em> progress and converge faster. My experiments are by no means conclusive. </p>
  
  <p>I would thence ask if it is known which one to perform better, converge faster or has smoother progress, in a wider range of tasks and settings?</p>
  
Tags: <reinforcement-learning><dqn><deep-rl>
Title: In DQN, updating target network every N steps or slowly update every step is better?
UserDisplayName: Phizaz
ThreadText: # In DQN, updating target network every N steps or slowly update every step is better?
  
  (Asked by: username_0 on 2019-02-28)
  
  The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with *semi-gradient* objectives. In Deep Q network, semi-gradient TD is used and with experience replay the training could diverge.
  
  Target network is a slow changing network designed to slowly track the main value network. In [Mnih 2013](https://arxiv.org/abs/1312.5602), it was designed to match the main network every $N$ steps. There is another way which slowly updates the weight in the direction to match the main network every step. To someone, the latter is called *Polyak updates*.
  
  I have done some very limited experiments and seen that with the same update rate, e.g. $N=10$, Polyak update would update with the rate of 0.1, I usually see Polyak updates to give *smoother* progress and converge faster. My experiments are by no means conclusive.
  
  I would thence ask if it is known which one to perform better, converge faster or has smoother progress, in a wider range of tasks and settings?
  
  **Tags:** reinforcement-learning, dqn, deep-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-28. Score: 2
  
  Most of papers and my experience support hard update, once per N steps. N is usually very bug, range in between 10^4 to 10^6. DQN training is slow. But that depend on the problem. If you DQN converge with soft update with weight ~.1 (N~10) your problem could be very simple.
  
  > **Comments:**
  > **username_0**: Would you mind put some more detailed research of yours? I would like to see something like this technique has been used in 2019 by the SOTA or so. Thanks.  
  > **username_1**: I was doing DQN for car driving simulation. The main points: even thogh DQN is off policy it's effectively act similar to on-policy due to overfitting on latest samples(latest measired in millions). That could be helped by increasing target net update intervaland to some degree size of replay buffer. Its very important to randomly remix samples from replay buffer then assembling training minibatch. Overfitting causing DQN drift, wich is main enemy  
  
  ---
  
  