Answers.Body: <p>DQN is a off-policy method, this means that you are fine (apart from some variance factor) with sampling data from another distribution</p>
  <p>However, Replay Buffer have a state distribution that is different from the new policy one, which might create some problem</p>
  <p>SGD on the other hand, with such method, has the problem of the fact that its convergence guarantees assumes that the samples are IID, which is not true in case of replay buffer (and generally, in most RL scenarios)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-28T13:06:37.230Z
Answers.LastActivityDate: 2023-09-28T13:06:37.230Z
Answers.UserDisplayName: Alberto
Body: <p>In Deep Q Learning algorithm the convergence is generally achieved using smart tricks like the target network and the replay buffer.</p>
  <p>However there is one thing which is not clear to me. When the Q network is trained through SGD, the loss function is an expectation over all possible states, which is approximated stochastically using samples from the replay buffer. But the replay buffer itself is not constructed from a unique stationary distribution: the behaviour policy used to collect state transitions typically changes during the overall training procedure (it typically becomes more greedy as new data are collected).</p>
  <p>As a result, we draw sample transitions from the replay buffer hoping to obtain a useful stochastic approximation of some stationary distribution, but those data were not actually drawn from a stationary distribution.</p>
  <p>Why isn't this an issue for the SGD procedure?</p>
  
Tags: <dqn><stochastic-gradient-descent>
Title: DQN Loss function - doubt about stochastic approximation
UserDisplayName: Ftoso91
ThreadText: # DQN Loss function - doubt about stochastic approximation
  
  (Asked by: username_0 on 2023-09-28)
  
  In Deep Q Learning algorithm the convergence is generally achieved using smart tricks like the target network and the replay buffer.
  
  However there is one thing which is not clear to me. When the Q network is trained through SGD, the loss function is an expectation over all possible states, which is approximated stochastically using samples from the replay buffer. But the replay buffer itself is not constructed from a unique stationary distribution: the behaviour policy used to collect state transitions typically changes during the overall training procedure (it typically becomes more greedy as new data are collected).
  
  As a result, we draw sample transitions from the replay buffer hoping to obtain a useful stochastic approximation of some stationary distribution, but those data were not actually drawn from a stationary distribution.
  
  Why isn't this an issue for the SGD procedure?
  
  **Tags:** dqn, stochastic-gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-28. Score: 0
  
  DQN is a off-policy method, this means that you are fine (apart from some variance factor) with sampling data from another distribution
  
  However, Replay Buffer have a state distribution that is different from the new policy one, which might create some problem
  
  SGD on the other hand, with such method, has the problem of the fact that its convergence guarantees assumes that the samples are IID, which is not true in case of replay buffer (and generally, in most RL scenarios)
  
  
  ---
  
  