Answers.Body: <p>Q-learning - which DQN is based on - is an off-policy reinforcement learning (RL) method. That means it can learn a <em>target policy</em> of optimal control whilst acting using a different <em>behaviour policy</em>. In addition, provided you use single step learning (as opposed to n-step or Q(<span class="math-container">$\lambda$</span>) variants), you don't need to worry much about the details of the behaviour policy. It is more efficient to learn from behaviour policies closer to the current best guess at optimal, but possible to learn from almost anything, including random behaviour.</p>
  
  <p>So it doesn't really matter too much if you change the behaviour during training.</p>
  
  <p>In your case, the script is actually doing more than just changing the behaviour. It is consistently filtering out state/action pairs that you have decided should never be taken, as a domain expert. This has two major consequences:</p>
  
  <ul>
  <li><p>It reduces the search space by whatever fraction of state/actions are now denied by your safety script.</p></li>
  <li><p>It prevents the agent from ever learning about certain state/action pairs, as they are never experienced.</p></li>
  </ul>
  
  <p>The first point means that your learning should in theory be more efficient. As for how much, you will have to try and see. It might only be a small amount if the problem states and actions are unlikely to be reached during exploration from near-optimal behaviour.</p>
  
  <p>The second point means that your agent will never learn by itself to avoid the problem state/action combinations. So you will always need to use the safety script.</p>
  
  <p>In fact you can view the safety script as a modification to the environment (as opposed to a modification to the agent), if its decisions are strict and consistent. Filtering available actions is a standard mechanism in RL when action space may vary depending on state.</p>
  
  <blockquote>
    <p>On which action should I perform backprop? </p>
  </blockquote>
  
  <p>In DQN, you don't "perform backprop" on an action. Instead you either use directly or store some observed data about the step: <span class="math-container">$s, a, r, s'$</span> where <span class="math-container">$s$</span> is the start state, <span class="math-container">$a$</span> the action taken, <span class="math-container">$r$</span> the immediate reward, and <span class="math-container">$s'$</span> the resulting state. You then update the the current action value estimate(s) based on a TD target <span class="math-container">$\delta = r + \gamma \text{max}_{a'} Q(s,a')$</span> either online or from the experience table.</p>
  
  <p>When Q-learning learns, it updates the estimate for the action value <span class="math-container">$Q(s, a)$</span> - and <span class="math-container">$a$</span> is taken from the actual behaviour (otherwise the agent will update its estimate of an action that it didn't take). So your answer here is to use - or more likely store in the experience table - the action actually taken. If the action recommended as optimal at that time is different, ignore the difference, what matters is the observed experience.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-11T14:36:04.630Z
Answers.Text: I don't know if it is appropriate to ask about this here in the comments, but what do you think about using Duelling architecture in this situation? I thought that maybe decoupling state-evaluation and action-evaluation may help, since we have an external controller. what do you think?
Answers.UserDisplayName: olinarr
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-11T14:51:26.713Z
Answers.Text: @NetHacker: I don't know about the dueling architecture, so cannot answer here. I think anyway that it is a different enough topic that you should ask a new question. The general rule is that comments are for clarifying posts or suggesting corrections to them. Follow-up questions with significant new details should ideally be new questions on the site
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-08T18:19:09.347Z
Answers.LastActivityDate: 2019-05-08T18:19:09.347Z
Answers.UserDisplayName: Neil Slater
Body: <p><strong>Background</strong></p>
  
  <p>I am working on a robotic arm controlled by a DQN + a python script I wrote.
  The DQN receives the 5 joint states, the coordinates of a target, the coordinates of the obstacle and outputs the best action to take (in terms of joint rotations).
  The python script checks if the action suggested by the DQN is safe. If it is, it performs it. Otherwise, it performs the second highest-ranking action from the DQN; and so on. If no action is possible, collision: we fail.</p>
  
  <p>During training, this python functionality wasn't present: the arm learned how to behave without anything else to correct his behaviour. With this addition on the already-trained network, the performance raised from 78 to 95%.
  Now my advisor (bachelor's thesis) asked me to leave the external controller on during training to check whether this improves learning.</p>
  
  <p><strong>Question</strong></p>
  
  <p>Here's what happens during training; at each step:</p>
  
  <ol>
  <li>the ANN selects an action</li>
  <li>if it is legal, the python script executes it, otherwise it chooses another one.</li>
  </ol>
  
  <p>Now... On which action should I perform backprop? The one proposed by the arm or the one which was really executed? (So, which action should I perform backprop on?)</p>
  
  <p>I am really confused. On the one hand, the arm <strong>did choose</strong> an action so my idea was that we should, in fact, learn on THAT action. On the other hand, during the exploration phase (<span class="math-container">$\epsilon$</span> greedy), we backprop on the action which was randomly selected and executed, with no interest on what was the output of the arm. So, it would be rational too, in this case, to perform backprop on the action really executed; so the one chosen by the script.</p>
  
  <p><strong>What is the right thing to do here?</strong>. <em>(Bonus question: is it reasonable to train with this functionality on? Wouldn't it be better to train the Network by itself, and then later, enhance its performance with this functionality?)</em></p>
  
Tags: <reinforcement-learning>
Title: DQN Agent helped by a controller: on which action should I perform backprop?
UserDisplayName: olinarr
ThreadText: # DQN Agent helped by a controller: on which action should I perform backprop?
  
  (Asked by: username_0 on 2019-05-08)
  
  **Background**
  
  I am working on a robotic arm controlled by a DQN + a python script I wrote.
  The DQN receives the 5 joint states, the coordinates of a target, the coordinates of the obstacle and outputs the best action to take (in terms of joint rotations).
  The python script checks if the action suggested by the DQN is safe. If it is, it performs it. Otherwise, it performs the second highest-ranking action from the DQN; and so on. If no action is possible, collision: we fail.
  
  During training, this python functionality wasn't present: the arm learned how to behave without anything else to correct his behaviour. With this addition on the already-trained network, the performance raised from 78 to 95%.
  Now my advisor (bachelor's thesis) asked me to leave the external controller on during training to check whether this improves learning.
  
  **Question**
  
  Here's what happens during training; at each step:
  
  1. the ANN selects an action
  2. if it is legal, the python script executes it, otherwise it chooses another one.
  
  Now... On which action should I perform backprop? The one proposed by the arm or the one which was really executed? (So, which action should I perform backprop on?)
  
  I am really confused. On the one hand, the arm **did choose** an action so my idea was that we should, in fact, learn on THAT action. On the other hand, during the exploration phase ($\epsilon$ greedy), we backprop on the action which was randomly selected and executed, with no interest on what was the output of the arm. So, it would be rational too, in this case, to perform backprop on the action really executed; so the one chosen by the script.
  
  **What is the right thing to do here?**. *(Bonus question: is it reasonable to train with this functionality on? Wouldn't it be better to train the Network by itself, and then later, enhance its performance with this functionality?)*
  
  **Tags:** reinforcement-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-08. Score: 2 | Accepted answer
  
  Q-learning - which DQN is based on - is an off-policy reinforcement learning (RL) method. That means it can learn a *target policy* of optimal control whilst acting using a different *behaviour policy*. In addition, provided you use single step learning (as opposed to n-step or Q($\lambda$) variants), you don't need to worry much about the details of the behaviour policy. It is more efficient to learn from behaviour policies closer to the current best guess at optimal, but possible to learn from almost anything, including random behaviour.
  
  So it doesn't really matter too much if you change the behaviour during training.
  
  In your case, the script is actually doing more than just changing the behaviour. It is consistently filtering out state/action pairs that you have decided should never be taken, as a domain expert. This has two major consequences:
  
  * It reduces the search space by whatever fraction of state/actions are now denied by your safety script.
  * It prevents the agent from ever learning about certain state/action pairs, as they are never experienced.
  
  The first point means that your learning should in theory be more efficient. As for how much, you will have to try and see. It might only be a small amount if the problem states and actions are unlikely to be reached during exploration from near-optimal behaviour.
  
  The second point means that your agent will never learn by itself to avoid the problem state/action combinations. So you will always need to use the safety script.
  
  In fact you can view the safety script as a modification to the environment (as opposed to a modification to the agent), if its decisions are strict and consistent. Filtering available actions is a standard mechanism in RL when action space may vary depending on state.
  
  > On which action should I perform backprop?
  
  In DQN, you don't "perform backprop" on an action. Instead you either use directly or store some observed data about the step: $s, a, r, s'$ where $s$ is the start state, $a$ the action taken, $r$ the immediate reward, and $s'$ the resulting state. You then update the the current action value estimate(s) based on a TD target $\delta = r + \gamma \text{max}\_{a'} Q(s,a')$ either online or from the experience table.
  
  When Q-learning learns, it updates the estimate for the action value $Q(s, a)$ - and $a$ is taken from the actual behaviour (otherwise the agent will update its estimate of an action that it didn't take). So your answer here is to use - or more likely store in the experience table - the action actually taken. If the action recommended as optimal at that time is different, ignore the difference, what matters is the observed experience.
  
  > **Comments:**
  > **username_0**: I don't know if it is appropriate to ask about this here in the comments, but what do you think about using Duelling architecture in this situation? I thought that maybe decoupling state-evaluation and action-evaluation may help, since we have an external controller. what do you think?  
  > **username_1**: @NetHacker: I don't know about the dueling architecture, so cannot answer here. I think anyway that it is a different enough topic that you should ask a new question. The general rule is that comments are for clarifying posts or suggesting corrections to them. Follow-up questions with significant new details should ideally be new questions on the site  
  
  ---
  
  