Body: <p>I have started working on Reinforcement Learning, specifically DQN. And I have watched some interesting videos on it. However, I have some doubts about how the model works.</p>
  <p>Let's say we are playing Atari Breakout where we have only 3 actions: left, stay still, right. We have 2 networks- the policy_net and the target_net (technically both are the same) and they give 3 outputs which are the q values for the 3 actions. During exploration we choose:</p>
  <pre><code>random.randrange(3)
  </code></pre>
  <p>and during exploitation, we choose:</p>
  <pre><code>argmax(policy_net output)
  </code></pre>
  <p>where the input of the policy net is the current state.</p>
  <p>Now, during each timeline of each episode, we are storing current_state, action, reward, and next_state in storage that we will later randomly shuffle and use in training, which we call experience replay memory. During training, let's say we extract a batch of (current_states, actions, rewards, next_states). Now, we get current_q_val and next_q_val as:</p>
  <pre><code>current_q_values = policy_net(current_states)
  next_q_values = target_net(next_states)
  </code></pre>
  <p>We use the following equation to find the loss:</p>
  <p><span class="math-container">$$q_*(s,a) - q(s,a) = \text{loss}$$</span></p>
  <p><span class="math-container">$$E[R_{t+1} + \gamma \max_{a'} q_*(s', a')] - E[\Sigma_{k=0}^ \inf \gamma^k R_{t+k+1}] = \text{loss}$$</span></p>
  <p>And for that, we find which one from the next_q_val is the best one and that we call <span class="math-container">$\max q_*(s',a')$</span> (we already have <span class="math-container">$q(s,a)=\text{current_q_values}$</span>).</p>
  <p>Now my question is, which is <span class="math-container">$R_{t+1}$</span> here? As we are taking a random batch from the experience replay memory, we don't have any specific time here and we cannot calculate the <span class="math-container">$R_{t+1}$</span> from the time <span class="math-container">$t$</span>. And if we simply use <span class="math-container">$q*(s,a) - q(s,a)$</span> or <span class="math-container">$\text{next_q_val} - \text{current_q_val}$</span>, then where is the importance of the reward? I don't really understand how we are using the rewards in the training. I mean, where are we making sure the positive and negative influences of good and bad rewards respectively? The fact that the agent takes an action (randomly or from the policy_net) which then gives a reward, I don't understand how to use this reward in the loss function so as to influence how the agent should take action given a state.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T08:46:12.823Z
Comments.Text: Where did you get the loss function from? The equation you give is a kind of conceptual description of "regret" in RL, and not used as a loss function in DQN in any meaningful way.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T10:35:54.240Z
Comments.Text: Hi @NeilSlater, I got the loss function from one of the youtube series on RL I watched. can you tell me what should be the corrected loss function? In the video series, they are using MSE loss on current_q_value and next_q_value
Comments.UserDisplayName: Khabbab Zakaria
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T11:02:36.900Z
Comments.Text: Hello. As I asked you on Stack Overflow, please, edit your post to format the math symbols with mathjax. You need to wrap them with `$` on either side, like this `$\pi$` to produce $\pi$.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T11:13:35.577Z
Comments.Text: "From one of the youtube series" is too vague for me to correct it. Either the video is wrong or your understanding of it is wrong. Using MSE loss for the neural network versus the TD targets calculated from each minibatch drives the learning, but it is important to note that is a moving target. It is possible for the MSE loss to get larger, but the agent behaviour to improve at the same time, at least in the short term. That is because the MSE loss does not capture the full RL objective, just the NN's role's objective.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T12:54:12.040Z
Comments.Text: @nbro, okay I changed the equation
Comments.UserDisplayName: Khabbab Zakaria
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-18T12:57:50.953Z
Comments.Text: @NeilSlater, yes I understand what you are saying. So, I was thinking of updating the target_net every 4 epochs while policy_net is getting updated every epoch, to make sure we are not always making it 'chasing its own tail'. But, I don't know how to add the rewards in the loss function. WHat I thought of is at each run we are getting currentstate, action, reward, nextstate. do we need to add this reward with the nextqvalue? and what should be the gamma then?
Comments.UserDisplayName: Khabbab Zakaria
Tags: <reinforcement-learning><python><dqn>
Title: How rewards are playing role in Deep Q Network
UserDisplayName: Khabbab Zakaria
ThreadText: # How rewards are playing role in Deep Q Network
  
  (Asked by: username_0 on 2021-11-18)
  
  I have started working on Reinforcement Learning, specifically DQN. And I have watched some interesting videos on it. However, I have some doubts about how the model works.
  Let's say we are playing Atari Breakout where we have only 3 actions: left, stay still, right. We have 2 networks- the policy_net and the target_net (technically both are the same) and they give 3 outputs which are the q values for the 3 actions. During exploration we choose:
  random.randrange(3)
  
  and during exploitation, we choose:
  argmax(policy_net output)
  
  where the input of the policy net is the current state.
  Now, during each timeline of each episode, we are storing current_state, action, reward, and next_state in storage that we will later randomly shuffle and use in training, which we call experience replay memory. During training, let's say we extract a batch of (current_states, actions, rewards, next_states). Now, we get current_q_val and next_q_val as:
  current_q_values = policy_net(current_states)
  next_q_values = target_net(next_states)
  
  We use the following equation to find the loss:
  $$q_*(s,a) - q(s,a) = \text{loss}$$
  $$E[R_{t+1} + \gamma \max_{a'} q_*(s', a')] - E[\Sigma_{k=0}^ \inf \gamma^k R_{t+k+1}] = \text{loss}$$
  And for that, we find which one from the next_q_val is the best one and that we call $\max q_*(s',a')$ (we already have $q(s,a)=\text{current_q_values}$).
  Now my question is, which is $R_{t+1}$ here? As we are taking a random batch from the experience replay memory, we don't have any specific time here and we cannot calculate the $R_{t+1}$ from the time $t$. And if we simply use $q*(s,a) - q(s,a)$ or $\text{next_q_val} - \text{current_q_val}$, then where is the importance of the reward? I don't really understand how we are using the rewards in the training. I mean, where are we making sure the positive and negative influences of good and bad rewards respectively? The fact that the agent takes an action (randomly or from the policy_net) which then gives a reward, I don't understand how to use this reward in the loss function so as to influence how the agent should take action given a state.
  
  **Tags:** reinforcement-learning, python, dqn
  > **Comments:**
  > **username_1**: Where did you get the loss function from? The equation you give is a kind of conceptual description of "regret" in RL, and not used as a loss function in DQN in any meaningful way.  
  > **username_0**: Hi @NeilSlater, I got the loss function from one of the youtube series on RL I watched. can you tell me what should be the corrected loss function? In the video series, they are using MSE loss on current_q_value and next_q_value  
  > **username_2**: Hello. As I asked you on Stack Overflow, please, edit your post to format the math symbols with mathjax. You need to wrap them with `$` on either side, like this `$\pi$` to produce $\pi$.  
  > **username_1**: "From one of the youtube series" is too vague for me to correct it. Either the video is wrong or your understanding of it is wrong. Using MSE loss for the neural network versus the TD targets calculated from each minibatch drives the learning, but it is important to note that is a moving target. It is possible for the MSE loss to get larger, but the agent behaviour to improve at the same time, at least in the short term. That is because the MSE loss does not capture the full RL objective, just the NN's role's objective.  
  > **username_0**: @nbro, okay I changed the equation  
  > **username_0**: @NeilSlater, yes I understand what you are saying. So, I was thinking of updating the target_net every 4 epochs while policy_net is getting updated every epoch, to make sure we are not always making it 'chasing its own tail'. But, I don't know how to add the rewards in the loss function. WHat I thought of is at each run we are getting currentstate, action, reward, nextstate. do we need to add this reward with the nextqvalue? and what should be the gamma then?  
  
  ---
  
  ## 0 answers
  
  