Answers.Body: <p>In DQN the loss function is designed to minimize the difference between the prediction network's <span class="math-container">$Q(s,a;\theta)$</span> and the target network's <span class="math-container">$Q^*(s,a)$</span> computed using the Bellman optimality equation as a conditional expectation defined on slide 39 of your reference. Since the target <span class="math-container">$Q^*(s,a)$</span> depends on the current state <span class="math-container">$s$</span>, the expectation is taken over the distribution of transitions <span class="math-container">$(s,a,r,s′)$</span> encountered during training from the replay buffer, thus the loss function inherently depends on the current state <span class="math-container">$s$</span>. The dependence on the current state reflects the idea that we want to learn a policy that is effective across the entire state space, not just for a specific state.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-01T07:55:11.460Z
Answers.LastActivityDate: 2024-09-01T07:55:11.460Z
Answers.UserDisplayName: cinch
Body: <p>Please see slide 39 in <a href="https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf" rel="nofollow noreferrer">https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf</a></p>
  <p>Why in loss function of DQN  does the expectation  depend on current state <span class="math-container">$s$</span>?</p>
  
Tags: <reinforcement-learning>
Title: Why in loss function of DQN does the expectation depend on current state $s$?
UserDisplayName: DSPinfinity
ThreadText: # Why in loss function of DQN does the expectation depend on current state $s$?
  
  (Asked by: username_0 on 2024-08-31)
  
  Please see slide 39 in <https://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture14.pdf>
  
  Why in loss function of DQN does the expectation depend on current state $s$?
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-01. Score: 2 | Accepted answer
  
  In DQN the loss function is designed to minimize the difference between the prediction network's $Q(s,a;\theta)$ and the target network's $Q^\*(s,a)$ computed using the Bellman optimality equation as a conditional expectation defined on slide 39 of your reference. Since the target $Q^\*(s,a)$ depends on the current state $s$, the expectation is taken over the distribution of transitions $(s,a,r,s′)$ encountered during training from the replay buffer, thus the loss function inherently depends on the current state $s$. The dependence on the current state reflects the idea that we want to learn a policy that is effective across the entire state space, not just for a specific state.
  
  
  ---
  
  