Answers.Body: <p>As you asked specifically about DQN, I will talk about Q-Learning. Since it is an off-policy algorithm, we can collect data with any exploratory policy <span class="math-container">$\beta$</span> whilst learning about the greedy policy <span class="math-container">$\pi(s) = \arg\max_aQ(s,a)$</span>. Typically <span class="math-container">$\beta$</span> will be an <span class="math-container">$\epsilon$</span>-greedy, but you do not want to use this when updating your DQN. That is because you update your DQN using the Q-Learning update rule, by training it to approximate <span class="math-container">$r(s, a) + \gamma \max_a Q_{\bar{\theta}}(s', a)$</span>, where <span class="math-container">$\bar{\theta}$</span> are the parameters of the target network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-02T23:24:45.137Z
Answers.Text: thx man it makes sense
Answers.UserDisplayName: Stef
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-02T21:24:06.847Z
Answers.LastActivityDate: 2023-07-02T21:24:06.847Z
Answers.UserDisplayName: David
Body: <p>In reinforcement learning (DQN) do I use epsilon when I am collecting examples from the environment or do I use epsilon when I am training the Q network and Target network ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-02T00:12:45.470Z
Comments.Text: When taking actions in the environment, assuming you’re using an off-policy algorithm like Q-Learning which DQN is based on.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-02T08:41:53.537Z
Comments.Text: I can write into a proper answer if you find the comment useful?
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-02T16:57:50.163Z
Comments.Text: yea I would find it very usefull
Comments.UserDisplayName: Stef
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-03T10:46:50.637Z
Comments.Text: People can probably guess what "epsilon" you're referring to (me included), but you should have provided more context. Although there's already an answer, I would still suggest that you edit your post and describe what you mean by "epsilon".
Comments.UserDisplayName: nbro
Tags: <neural-networks><reinforcement-learning><deep-learning><training><epsilon-greedy-policy>
Title: When do you use epsilon in Reinforcement Learning?
UserDisplayName: Stef
ThreadText: # When do you use epsilon in Reinforcement Learning?
  
  (Asked by: username_0 on 2023-07-01)
  
  In reinforcement learning (DQN) do I use epsilon when I am collecting examples from the environment or do I use epsilon when I am training the Q network and Target network ?
  
  **Tags:** neural-networks, reinforcement-learning, deep-learning, training, epsilon-greedy-policy
  
  > **Comments:**
  > **username_1**: When taking actions in the environment, assuming you’re using an off-policy algorithm like Q-Learning which DQN is based on.  
  > **username_1**: I can write into a proper answer if you find the comment useful?  
  > **username_0**: yea I would find it very usefull  
  > **username_2**: People can probably guess what "epsilon" you're referring to (me included), but you should have provided more context. Although there's already an answer, I would still suggest that you edit your post and describe what you mean by "epsilon".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-02. Score: 1 | Accepted answer
  
  As you asked specifically about DQN, I will talk about Q-Learning. Since it is an off-policy algorithm, we can collect data with any exploratory policy $\beta$ whilst learning about the greedy policy $\pi(s) = \arg\max\_aQ(s,a)$. Typically $\beta$ will be an $\epsilon$-greedy, but you do not want to use this when updating your DQN. That is because you update your DQN using the Q-Learning update rule, by training it to approximate $r(s, a) + \gamma \max\_a Q\_{\bar{\theta}}(s', a)$, where $\bar{\theta}$ are the parameters of the target network.
  
  > **Comments:**
  > **username_0**: thx man it makes sense  
  
  ---
  
  