Answers.Body: <p>You can determine the value of <code>epsilon</code> during training, for example, by using a <em>linear decay schedule</em> as a function of the steps or every <code>T</code> of them: you start from an <em>initial epsilon</em> (e.g. <span class="math-container">$\epsilon_0 = 1$</span>) and decay with some factor (e.g. <span class="math-container">$\gamma_\epsilon = 1/T$</span>) until a minimum/final value of epsilon is reached (e.g. <span class="math-container">$\epsilon_\min = 0.1$</span>).</p>
  <p>The rule is something like:
  <span class="math-container">$$\epsilon \leftarrow \max\big(\epsilon_\min, \epsilon - (\epsilon_0 - \epsilon_\min) * \gamma_\epsilon\big) $$</span></p>
  <p>In practice you start with high epsilon so that your agent explores a lot initially, then you start to decrease it gradually until it reaches a minimum value, after that your agent mostly exploits (indeed if final <span class="math-container">$\epsilon=0$</span> that it only exploits.)</p>
  <p>Moreover, if you want to periodically increase <span class="math-container">$\epsilon$</span> but not to the max value each time, you can decay <span class="math-container">$\epsilon_0$</span> too: for example, you can <span class="math-container">$\epsilon_0\leftarrow \epsilon_0*\gamma_\epsilon$</span> every <code>T</code> steps and then proceed with the linear decay. It would be like resetting the epsilon from time to time, and then gradually decrease that to favor exploitation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-03T20:31:09.100Z
Answers.LastActivityDate: 2023-05-03T21:06:12.870Z
Answers.LastEditDate: 2023-05-03T21:06:12.870Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to train it again but not from zero. I need to change just epsilon to discover some new states in a way that I don't lose my previous experiences. In other words, how to increase epsilon but not too much? I don't want a fixed epsilon_0 like 1 or 0.99.</p>
  
Tags: <reinforcement-learning><dqn><exploration-exploitation-tradeoff><epsilon-greedy-policy><exploration-strategies>
Title: In DQN, how to increase epsilon but not too much?
UserDisplayName: Mouad
ThreadText: # In DQN, how to increase epsilon but not too much?
  
  (Asked by: username_0 on 2023-05-03)
  
  I am using DQN algorithm in a non-stationary problem in a continuous learning. My environment gives me some new states each T steps. For example after 10 000 steps, I get some new states and I need to train it again but not from zero. I need to change just epsilon to discover some new states in a way that I don't lose my previous experiences. In other words, how to increase epsilon but not too much? I don't want a fixed epsilon\_0 like 1 or 0.99.
  
  **Tags:** reinforcement-learning, dqn, exploration-exploitation-tradeoff, epsilon-greedy-policy, exploration-strategies
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-03. Score: 0 | Accepted answer
  
  You can determine the value of `epsilon` during training, for example, by using a *linear decay schedule* as a function of the steps or every `T` of them: you start from an *initial epsilon* (e.g. $\epsilon\_0 = 1$) and decay with some factor (e.g. $\gamma\_\epsilon = 1/T$) until a minimum/final value of epsilon is reached (e.g. $\epsilon\_\min = 0.1$).
  
  The rule is something like:
  $$\epsilon \leftarrow \max\big(\epsilon\_\min, \epsilon - (\epsilon\_0 - \epsilon\_\min) \* \gamma\_\epsilon\big) $$
  
  In practice you start with high epsilon so that your agent explores a lot initially, then you start to decrease it gradually until it reaches a minimum value, after that your agent mostly exploits (indeed if final $\epsilon=0$ that it only exploits.)
  
  Moreover, if you want to periodically increase $\epsilon$ but not to the max value each time, you can decay $\epsilon\_0$ too: for example, you can $\epsilon\_0\leftarrow \epsilon\_0\*\gamma\_\epsilon$ every `T` steps and then proceed with the linear decay. It would be like resetting the epsilon from time to time, and then gradually decrease that to favor exploitation.
  
  
  ---
  
  