Answers.Body: <p>You have the concept slightly wrong.</p>
  
  <p>This part is mostly correct:</p>
  
  <blockquote>
    <p>In Reinforcement learning a random-initialized network will first "play"/"do" a sequence of moves in an environment. (In this case a Game). After that, it will receive a reward r. </p>
  </blockquote>
  
  <p>Technically neural networks are not required in RL, and it is really worth studying some simple systems that don't need them. It will make everything much clearer.</p>
  
  <p>A reward <span class="math-container">$r$</span> can be received on every time step. However, some environments will only have a single reward at the end for success or failure for a whole episode - e.g. an instance of a game like chess where a player wins or loses.</p>
  
  <p>This part is where things go a bit off track:</p>
  
  <blockquote>
    <p>Furthermore a q-Value gets defined by the [developer]. This reward times the q-Value q to the power of the position n of the action will be [fed] back using BP.</p>
  </blockquote>
  
  <p>Q values are one type of data that can be calculated for an agent acting in a <a href="https://en.wikipedia.org/wiki/Markov_decision_process" rel="nofollow noreferrer">Markov Decision Process</a>. They are also called "action values" and they are not usually defined by a developer. The q value, <em>if correct</em> should return the expected future sum of rewards from following a current policy. One way of writing this is:</p>
  
  <p><span class="math-container">$$q(s,a) = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}| S_t=s, A_t=a]$$</span></p>
  
  <p>In natural language, the q value for state s and action a is the expected value (when following the given policy) of the discounted sum of rewards, starting from the given state and action. The discount factor, <span class="math-container">$\gamma$</span> can take any value from <span class="math-container">$0$</span> up to <span class="math-container">$1$</span>, but only strictly episodic problems (which always terminate) should use the value <span class="math-container">$1$</span></p>
  
  <p>A developer does not get to define that (except they might get to choose reward system and value of <span class="math-container">$\gamma$</span>). Instead, they need to implement <em>something</em> that estimates the value of <span class="math-container">$q(s,a)$</span> based on what the agent has experienced. There are a few different algorithms that can do this. A popular one is called Q learning.</p>
  
  <p>Regarding "[fed] back using BP", this is correct if you are using a neural network. Typically in DQN (Q learning with neural networks), this just consists of creating a small sample of training data from recent experience and training the neural network almost identically to supervised learning.</p>
  
  <blockquote>
    <p>So how do I know how slight chances in <span class="math-container">$\vec{w}$</span> are changing <span class="math-container">$rq^n$</span>?</p>
  </blockquote>
  
  <p>Definitely don't use <span class="math-container">$rq^n$</span> - there is no purpose to that quantity in RL. Instead for value-based RL, you are mostly interested in your estimate for Q value. This might be written <span class="math-container">$\hat{q}(s,a,\vec{w}) \approx q(s,a)$</span>.</p>
  
  <p>However, in general your question stands. If you have implemented a neural network to learn q values, how do you know if it is working?</p>
  
  <p>There are actually two parts to this problem:</p>
  
  <ul>
  <li><p>How do you know whether the agent is getting better at it's task?</p></li>
  <li><p>How do you know whether the q values are getting more accurate?</p></li>
  </ul>
  
  <p>What you need to do is measure, and maybe plot some relevant quantities. </p>
  
  <p>For the first question, you would typically plot the total reward that the agent gets each episode. This will be noisy, so it is a good idea to smooth it out by taking some kind of moving average (e.g. average total reward over last 100 episodes).</p>
  
  <p>For the second question, it is normal to plot some loss function of the network, just like supervised learning. Typically this is Mean Squared Error loss, as the network is learning a regression to predict q values given <span class="math-container">$s$</span> and <span class="math-container">$a$</span>.  You can compare observed sums of discounted reward (aka "return" or "utility") with the earlier predicted ones, and take the error function. You need to get some measure of a "true" value of q - usually a noisy sample taken during training or testing, and measure loss. For MSE that might be </p>
  
  <p><span class="math-container">$$J(\vec{w}) = \frac{1}{2|D|}\sum_{(s,a) \in D}(\hat{q}(s,a, \vec{w}) - q(s,a))^2$$</span></p>
  
  <p>Where <span class="math-container">$D$</span> is some dataset you have put together of <span class="math-container">$s,a$</span> and <span class="math-container">$q(s,a)$</span> measurements to test with. If this looks familiar to you from supervised learning MSE loss, then that's correct - it is essentially the same thing, just different how you go about collecting the data.</p>
  
  <p>You may expect the loss function for <span class="math-container">$\hat{q}$</span> in Q learning to be somewhat unstable as the agent learns. That's because in Q learning, the policy is updating at the same time as the estimates are improving. Which makes the estimates out-of-date. However, it should still be possible to see a reduction in error as learning progresses. If it becomes stable at a relatively low value compared to initially, then the agent has probably learned all that it can - although sometimes new discoveries by the agent can open up more improvements, even late in training, and throw the error function out again. </p>
  
  <p>Note that a low value of the error function does not mean you have an optimal agent. It means that the value function estimate is good for how the agent is currently behaving. In turn that means the agent cannot make further improvements without new and different experience.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-16T21:03:32.277Z
Answers.Text: Thanks for this for-me-deep-enough explanation. I certainly understood what was my fault! Although this mathematic is in the moment a little bit deep for me and I am not quite sure how to read the equations, I will certainly go one.
  
  Have an awesome day!
Answers.UserDisplayName: TVSuchty
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-16T17:03:08.297Z
Answers.LastActivityDate: 2018-10-16T17:24:57.423Z
Answers.LastEditDate: 2018-10-16T17:24:57.423Z
Answers.UserDisplayName: Neil Slater
Body: <p>I already know the basics of the basic of Machine Learning. E.g.: Backpropagation, Convolution, etc.</p>
  
  <p>First of let me explain Reinforcement learning to make sure I grasped the concept correctly.</p>
  
  <p>In Reinforcement learning a random-initialized network will first "play"/"do" a sequence of moves in an environment. (In this case a Game). After that, it will receive a reward <span class="math-container">$r$</span>. Furthermore a q-Value gets defined by the engineer/hooby coder. This reward times the q-Value <span class="math-container">$q$</span> to the power of the  position <span class="math-container">$n$</span> of the action will be feeded back using BP. </p>
  
  <p><strong>So how do I know how slight chances in <span class="math-container">$\vec{w}$</span> are changing <span class="math-container">$rq^n$</span>?</strong></p>
  
Tags: <reinforcement-learning><game-ai><backpropagation>
Title: How do I know how changes in the weights are changing the reward in Reinforcement Learning
UserDisplayName: TVSuchty
ThreadText: # How do I know how changes in the weights are changing the reward in Reinforcement Learning
  
  (Asked by: username_0 on 2018-10-16)
  
  I already know the basics of the basic of Machine Learning. E.g.: Backpropagation, Convolution, etc.
  
  First of let me explain Reinforcement learning to make sure I grasped the concept correctly.
  
  In Reinforcement learning a random-initialized network will first "play"/"do" a sequence of moves in an environment. (In this case a Game). After that, it will receive a reward $r$. Furthermore a q-Value gets defined by the engineer/hooby coder. This reward times the q-Value $q$ to the power of the position $n$ of the action will be feeded back using BP.
  
  **So how do I know how slight chances in $\vec{w}$ are changing $rq^n$?**
  
  **Tags:** reinforcement-learning, game-ai, backpropagation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-16. Score: 1
  
  You have the concept slightly wrong.
  
  This part is mostly correct:
  
  > In Reinforcement learning a random-initialized network will first "play"/"do" a sequence of moves in an environment. (In this case a Game). After that, it will receive a reward r.
  
  Technically neural networks are not required in RL, and it is really worth studying some simple systems that don't need them. It will make everything much clearer.
  
  A reward $r$ can be received on every time step. However, some environments will only have a single reward at the end for success or failure for a whole episode - e.g. an instance of a game like chess where a player wins or loses.
  
  This part is where things go a bit off track:
  
  > Furthermore a q-Value gets defined by the [developer]. This reward times the q-Value q to the power of the position n of the action will be [fed] back using BP.
  
  Q values are one type of data that can be calculated for an agent acting in a [Markov Decision Process](https://en.wikipedia.org/wiki/Markov_decision_process). They are also called "action values" and they are not usually defined by a developer. The q value, *if correct* should return the expected future sum of rewards from following a current policy. One way of writing this is:
  
  $$q(s,a) = \mathbb{E}\_{\pi}[\sum\_{k=0}^{\infty}\gamma^k R\_{t+k+1}| S\_t=s, A\_t=a]$$
  
  In natural language, the q value for state s and action a is the expected value (when following the given policy) of the discounted sum of rewards, starting from the given state and action. The discount factor, $\gamma$ can take any value from $0$ up to $1$, but only strictly episodic problems (which always terminate) should use the value $1$
  
  A developer does not get to define that (except they might get to choose reward system and value of $\gamma$). Instead, they need to implement *something* that estimates the value of $q(s,a)$ based on what the agent has experienced. There are a few different algorithms that can do this. A popular one is called Q learning.
  
  Regarding "[fed] back using BP", this is correct if you are using a neural network. Typically in DQN (Q learning with neural networks), this just consists of creating a small sample of training data from recent experience and training the neural network almost identically to supervised learning.
  
  > So how do I know how slight chances in $\vec{w}$ are changing $rq^n$?
  
  Definitely don't use $rq^n$ - there is no purpose to that quantity in RL. Instead for value-based RL, you are mostly interested in your estimate for Q value. This might be written $\hat{q}(s,a,\vec{w}) \approx q(s,a)$.
  
  However, in general your question stands. If you have implemented a neural network to learn q values, how do you know if it is working?
  
  There are actually two parts to this problem:
  
  * How do you know whether the agent is getting better at it's task?
  * How do you know whether the q values are getting more accurate?
  
  What you need to do is measure, and maybe plot some relevant quantities.
  
  For the first question, you would typically plot the total reward that the agent gets each episode. This will be noisy, so it is a good idea to smooth it out by taking some kind of moving average (e.g. average total reward over last 100 episodes).
  
  For the second question, it is normal to plot some loss function of the network, just like supervised learning. Typically this is Mean Squared Error loss, as the network is learning a regression to predict q values given $s$ and $a$. You can compare observed sums of discounted reward (aka "return" or "utility") with the earlier predicted ones, and take the error function. You need to get some measure of a "true" value of q - usually a noisy sample taken during training or testing, and measure loss. For MSE that might be
  
  $$J(\vec{w}) = \frac{1}{2|D|}\sum\_{(s,a) \in D}(\hat{q}(s,a, \vec{w}) - q(s,a))^2$$
  
  Where $D$ is some dataset you have put together of $s,a$ and $q(s,a)$ measurements to test with. If this looks familiar to you from supervised learning MSE loss, then that's correct - it is essentially the same thing, just different how you go about collecting the data.
  
  You may expect the loss function for $\hat{q}$ in Q learning to be somewhat unstable as the agent learns. That's because in Q learning, the policy is updating at the same time as the estimates are improving. Which makes the estimates out-of-date. However, it should still be possible to see a reduction in error as learning progresses. If it becomes stable at a relatively low value compared to initially, then the agent has probably learned all that it can - although sometimes new discoveries by the agent can open up more improvements, even late in training, and throw the error function out again.
  
  Note that a low value of the error function does not mean you have an optimal agent. It means that the value function estimate is good for how the agent is currently behaving. In turn that means the agent cannot make further improvements without new and different experience.
  
  > **Comments:**
  > **username_0**: Thanks for this for-me-deep-enough explanation. I certainly understood what was my fault! Although this mathematic is in the moment a little bit deep for me and I am not quite sure how to read the equations, I will certainly go one.  Have an awesome day!  
  
  ---
  
  