Answers.Body: <p>What normally happens in DQN is the following:</p>
  
  <p>First, the NN is used to estimate an approximation of Q for each state-action pair by using a vector of weights:
  <span class="math-container">$$Q(s,a,w) \simeq Q(s,a)$$</span></p>
  
  <p>There are two options: You can either feed it a certain state, and get as output the value of each possible action, or, you can input a state-action pair and get as output the value of that pair. (basically the NN gives you the policy)</p>
  
  <p>After having this output you can now use your environment to simulate the next state. So basically you apply the action given by the NN (maybe using <span class="math-container">$\epsilon -greedy$</span>), and you get the next state, and your next reward from your environment by observing it.</p>
  
  <p>Where does the Q function comes in all of this? The Q-function 
  <span class="math-container">$$Q(s,a) = Q(s,a) + \alpha [r(s,a) + \gamma  max_a{Q(s',a)} - Q(s,a)]$$</span>
  that in standard Q-learning is used to update the values after each simulation step, is now used as the Target of the neural network, to make the back propagation.</p>
  
  <p>There are some considerations you have to make to help this converge, mainly experience replay and fixed Q-targets.</p>
  
  <p>So when you say " I don't have yet the next state", is because you need to simulate the action that is selected to get this next state.</p>
  
  <p><strong>EDIT:</strong> The main thing you might be missing is that the right part of the equation you present in the title is called TD-Target, and it is not used to make the updates, you might see it as the final result you want to achieve <span class="math-container">$Q^*(s,a)$</span> (the optimal values, after convergence). But to make the update you need to use the weighted function with the previous value of <span class="math-container">$Q(s,a)$</span> as I showed above, using a learning rate <span class="math-container">$0&lt; \alpha &lt; 1$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-16T10:09:09.837Z
Answers.LastActivityDate: 2019-05-16T11:03:40.107Z
Answers.LastEditDate: 2019-05-16T11:03:40.107Z
Answers.UserDisplayName: Miguel Saraiva
Body: <p>Ok, due to <a href="https://ai.stackexchange.com/questions/12318/drone-training-how-to-train-without-training-data">previous question</a> I was pointed to use reinfrocement learning.</p>
  
  <p>So far what I understood from random websites is the following:</p>
  
  <ul>
  <li>there is a Q(s,a) function involved</li>
  <li>I can assume my neural network ~ Q(s,a)</li>
  <li>my simulation has a state (N input variables)</li>
  <li>my actor can perform M possible actions (M output variables)</li>
  <li>at each step of the simulation my actor perform just the action corresponding to the <code>max(outputs)</code></li>
  <li>(in my case the actions are 1/2/3 % increase or decrease to propellers thrust force.)</li>
  </ul>
  
  <p><strong>From this <a href="https://neuro.cs.ut.ee/demystifying-deep-reinforcement-learning/" rel="nofollow noreferrer">website</a> I found that at some point I have to</strong>:</p>
  
  <ul>
  <li>Estimate outputs Q[t] (or so called q-values)</li>
  <li>Estimate outputs over next state Q[t+1]</li>
  <li>Let the backpropagation algorithm perform error correction only on the action performed on next state.</li>
  </ul>
  
  <p><strong>The last 3 points are not clear at all to me, infact I don't have yet the next state what I do instead is</strong>:</p>
  
  <ul>
  <li>Estimate previous outputs Q[t-1]</li>
  <li>Estimate current outputs Q[t]</li>
  <li>Let backpropagation fix the error for max q value only</li>
  </ul>
  
  <p>Actually for code I use just this library which is simple enough to allow me understand what happens inside:</p>
  
  <p><a href="https://github.com/Hagsten/NeuralNetwork/tree/master/NeuralNetwork" rel="nofollow noreferrer">NeuralNetwork library</a></p>
  
  <p>Initializing the neural network (with N input neurons, N+M hidden neurons and M output neurons) is as simple as</p>
  
  <pre><code>Network network = new NeuralNetwork( N, N+M, M);
  </code></pre>
  
  <p>Then I think to understand there is the need for an arbitrary reward function</p>
  
  <pre><code>public double R()
  {
       double distance = (currentPosition - targetPosition).VectorMagnitude();
       if(distance&lt;100)
           return 100-distance; // the nearest the greatest the reward
       return -1; // too far
  }
  </code></pre>
  
  <p>then what I do is:</p>
  
  <pre><code>// init step
  var previousInputs = ReadInputs();
  UpdateInputs();
  var currentInputs = ReadInputs();
  
  //Estimate previous outputs Q[t-1]
  previousOutputs = network.Query( previousInputs );
  
  //Estimate current outputs Q[t]
  currentOutputs = network.Query( currentInputs);
  
  // compute modified max value
  int maxIndex = 0;
  double maxValue = double.MinValue;
  SelectMax( currentOutputs, out maxValue, out maxIndex);
  
  // apply the modified max value to PREVIOUS outputs
  previousOutputs[maxIndex] = R() + discountValue* currentOutputs[maxIndex];
  
  //Let backpropagation fix the error for max q value only
  network.Train( previousInputs, previousOutputs);
  
  // advance simulation by 1 step and see what happens 
  RunPhysicsSimulationStep(1/200.0);
  DrawEverything();
  </code></pre>
  
  <p>But it doesn't seem to work very nice. I let simulation running for over one hour without success. Probably I'm reading the algorithm in a wrong way.</p>
  
Tags: <neural-networks><reinforcement-learning><q-learning>
Title: Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?
UserDisplayName: CoffeDeveloper
ThreadText: # Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?
  
  (Asked by: username_0 on 2019-05-15)
  
  Ok, due to [previous question](https://ai.stackexchange.com/questions/12318/drone-training-how-to-train-without-training-data) I was pointed to use reinfrocement learning.
  
  So far what I understood from random websites is the following:
  
  * there is a Q(s,a) function involved
  * I can assume my neural network ~ Q(s,a)
  * my simulation has a state (N input variables)
  * my actor can perform M possible actions (M output variables)
  * at each step of the simulation my actor perform just the action corresponding to the `max(outputs)`
  * (in my case the actions are 1/2/3 % increase or decrease to propellers thrust force.)
  
  **From this [website](https://neuro.cs.ut.ee/demystifying-deep-reinforcement-learning/) I found that at some point I have to**:
  
  * Estimate outputs Q[t] (or so called q-values)
  * Estimate outputs over next state Q[t+1]
  * Let the backpropagation algorithm perform error correction only on the action performed on next state.
  
  **The last 3 points are not clear at all to me, infact I don't have yet the next state what I do instead is**:
  
  * Estimate previous outputs Q[t-1]
  * Estimate current outputs Q[t]
  * Let backpropagation fix the error for max q value only
  
  Actually for code I use just this library which is simple enough to allow me understand what happens inside:
  
  [NeuralNetwork library](https://github.com/Hagsten/NeuralNetwork/tree/master/NeuralNetwork)
  
  Initializing the neural network (with N input neurons, N+M hidden neurons and M output neurons) is as simple as
  
  ```
  Network network = new NeuralNetwork( N, N+M, M);
  
  ```
  
  Then I think to understand there is the need for an arbitrary reward function
  
  ```
  public double R()
  {
       double distance = (currentPosition - targetPosition).VectorMagnitude();
       if(distance<100)
           return 100-distance; // the nearest the greatest the reward
       return -1; // too far
  }
  
  ```
  
  then what I do is:
  
  ```
  // init step
  var previousInputs = ReadInputs();
  UpdateInputs();
  var currentInputs = ReadInputs();
  
  //Estimate previous outputs Q[t-1]
  previousOutputs = network.Query( previousInputs );
  
  //Estimate current outputs Q[t]
  currentOutputs = network.Query( currentInputs);
  
  // compute modified max value
  int maxIndex = 0;
  double maxValue = double.MinValue;
  SelectMax( currentOutputs, out maxValue, out maxIndex);
  
  // apply the modified max value to PREVIOUS outputs
  previousOutputs[maxIndex] = R() + discountValue* currentOutputs[maxIndex];
  
  //Let backpropagation fix the error for max q value only
  network.Train( previousInputs, previousOutputs);
  
  // advance simulation by 1 step and see what happens 
  RunPhysicsSimulationStep(1/200.0);
  DrawEverything();
  
  ```
  
  But it doesn't seem to work very nice. I let simulation running for over one hour without success. Probably I'm reading the algorithm in a wrong way.
  
  **Tags:** neural-networks, reinforcement-learning, q-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-16. Score: 2 | Accepted answer
  
  What normally happens in DQN is the following:
  
  First, the NN is used to estimate an approximation of Q for each state-action pair by using a vector of weights:
  $$Q(s,a,w) \simeq Q(s,a)$$
  
  There are two options: You can either feed it a certain state, and get as output the value of each possible action, or, you can input a state-action pair and get as output the value of that pair. (basically the NN gives you the policy)
  
  After having this output you can now use your environment to simulate the next state. So basically you apply the action given by the NN (maybe using $\epsilon -greedy$), and you get the next state, and your next reward from your environment by observing it.
  
  Where does the Q function comes in all of this? The Q-function
  $$Q(s,a) = Q(s,a) + \alpha [r(s,a) + \gamma max\_a{Q(s',a)} - Q(s,a)]$$
  that in standard Q-learning is used to update the values after each simulation step, is now used as the Target of the neural network, to make the back propagation.
  
  There are some considerations you have to make to help this converge, mainly experience replay and fixed Q-targets.
  
  So when you say " I don't have yet the next state", is because you need to simulate the action that is selected to get this next state.
  
  **EDIT:** The main thing you might be missing is that the right part of the equation you present in the title is called TD-Target, and it is not used to make the updates, you might see it as the final result you want to achieve $Q^\*(s,a)$ (the optimal values, after convergence). But to make the update you need to use the weighted function with the previous value of $Q(s,a)$ as I showed above, using a learning rate $0< \alpha < 1$
  
  
  ---
  
  