Answers.Body: <blockquote>
    <p>Is this a sign that the algorithm diverged?</p>
  </blockquote>
  
  <p>It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to a design/architecture problem.</p>
  
  <p>Other common causes of algorithm failing with a fixed action choice include:</p>
  
  <ul>
  <li><p>Neural network inputs not scaled before use.</p></li>
  <li><p>Large reward values causing large initial squared errors (either re-scale rewards or reduce learning rate to fix)</p></li>
  <li><p>State representation too far from <a href="https://en.wikipedia.org/wiki/Markov_property" rel="noreferrer">Markov property</a> assumption</p></li>
  <li><p>A bug in code (almost anywhere, unfortunately)</p></li>
  <li><p><a href="https://en.wikipedia.org/wiki/Catastrophic_interference" rel="noreferrer">Catastrophic forgetting</a> due to focusing too much on specific results and generalising from them incorrectly. Your agent might be suffering from this if it starts to learn correctly, reaching some level of competence at the task before failing.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T12:18:23.397Z
Answers.Text: I'm in the "deadly triad" case, so it is not such a surprise that it can easily diverge. I still have to try more sophisticated reward functions so I hope that I can soft that tendency in this way.
Answers.UserDisplayName: unter_983
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-31T09:48:25.073Z
Answers.LastActivityDate: 2020-05-31T09:48:25.073Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. Sometimes it happens that, after a few steps, the algorithm starts choosing only one between the possible actions (apart from a few steps where I suppose it explores, given the epsilon-greedy policy it follows), leading to bad results in terms of cumulative rewards. Is this a sign that the algorithm diverged? </p>
  
Tags: <reinforcement-learning><q-learning><deep-rl><convergence>
Title: If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?
UserDisplayName: unter_983
ThreadText: # If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?
  
  (Asked by: username_0 on 2020-05-31)
  
  I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. Sometimes it happens that, after a few steps, the algorithm starts choosing only one between the possible actions (apart from a few steps where I suppose it explores, given the epsilon-greedy policy it follows), leading to bad results in terms of cumulative rewards. Is this a sign that the algorithm diverged?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, convergence
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-31. Score: 5 | Accepted answer
  
  > Is this a sign that the algorithm diverged?
  
  It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to a design/architecture problem.
  
  Other common causes of algorithm failing with a fixed action choice include:
  
  * Neural network inputs not scaled before use.
  * Large reward values causing large initial squared errors (either re-scale rewards or reduce learning rate to fix)
  * State representation too far from [Markov property](https://en.wikipedia.org/wiki/Markov_property) assumption
  * A bug in code (almost anywhere, unfortunately)
  * [Catastrophic forgetting](https://en.wikipedia.org/wiki/Catastrophic_interference) due to focusing too much on specific results and generalising from them incorrectly. Your agent might be suffering from this if it starts to learn correctly, reaching some level of competence at the task before failing.
  
  > **Comments:**
  > **username_0**: I'm in the "deadly triad" case, so it is not such a surprise that it can easily diverge. I still have to try more sophisticated reward functions so I hope that I can soft that tendency in this way.  
  
  ---
  
  