Answers.Body: <p>I will first explain <em>briefly</em> to you the difference between supervised learning and reinforcement learning to make sure that you don't have any misunderstandings. In supervised learning you are provided with some data <span class="math-container">$\{(\textbf{x}_i, y_i)\}_{i=1}^n$</span> where <span class="math-container">$\textbf{x}_i$</span> are the features for data point <span class="math-container">$i$</span> and <span class="math-container">$y_i$</span> is its true label. Now, the aim of supervised learning is to learn a function <span class="math-container">$f$</span> that can accurately predict the label of a data point given its features. In deep learning this function is a neural network, <span class="math-container">$f_\theta(\cdot)$</span>. To optimise the parameters we obtain the models prediction for the label of <span class="math-container">$y$</span>, denoted typically by <span class="math-container">$\hat{y} = f_\theta(x)$</span> and we look to optimise the parameters of the function <span class="math-container">$\theta$</span> by minimising the loss <span class="math-container">$\mathcal{L}(\hat{y}, y)$</span> (note that here the loss is a function <span class="math-container">$\mathcal{L}: \mathcal{Y} \times \mathcal{Y} \rightarrow \mathbb{R}$</span>.</p>
  <p>In reinforcement learning things are quite different. We are not provided with any data. Instead we have a <a href="https://en.wikipedia.org/wiki/Markov_decision_process" rel="nofollow noreferrer">Markov Decision Process (MDP)</a> and an environment that we can interact with. The state space is defined by the MDP and we can get samples, mainly tuples of the form <span class="math-container">$(s, a, r, s')$</span>, that we can use to teach our agent how to find an optimal policy - typically an optimal policy is one that takes an action given the current state that will maximise the sum of the future rewards of the episode. So in reinforcement you can probably see that things are quite different to supervised learning, mainly in the way that we 'obtain' our data.</p>
  <p>Of course these explanations are gross oversimplifications of the learning process in both paradigms and should only be used as a brief example to try to emphasise the different between the two learning paradigms.</p>
  <p>Now, there are two ways you can parameterise your Q-function using a neural network:</p>
  <ol>
  <li>The network takes as input the current state and outputs Q-values for each potential action; i.e. the output is a vector in <span class="math-container">$\mathbb{R}^{|\mathcal{A}|}$</span>;</li>
  <li>The network takes as input the state and action and outputs a real number which is the Q-value for the state, action tuple you pass as input to the network.</li>
  </ol>
  <p>The second way is usually reserved for instances where you have a huge action space but only ever would consider a few feasible actions at each state - this saves computational complexity.</p>
  <p>Now, to answer your question:</p>
  <p>In case 1) you are assumed to have access to a transition tuple <span class="math-container">$(s, a, r, s')$</span>. The temporal difference, which we will use as the target, is <span class="math-container">$\hat{y} = r + \max_a Q(s', a)$</span>. Now, as our output of <span class="math-container">$Q(s', \cdot)$</span> is an <span class="math-container">$\mathbb{R}^{|\mathcal{A}|}$</span> what we do is make a forward pass of the network for the current <span class="math-container">$(s, a)$</span> tuple, i.e. we get <span class="math-container">$x = Q(s, a)$</span>, and then change the element that corresponds to the action which satisfies <span class="math-container">$\arg\max_a Q(s', a)$</span> of <span class="math-container">$x$</span> to <span class="math-container">$\hat{y}$</span>, so we have now got our augmented input <span class="math-container">$\tilde{x}$</span> which serves as our target.</p>
  <p>To make that step a bit clearer, suppose we have a 2-dimensional action space and the <span class="math-container">$\arg\max_a Q(s', a)$</span> is the action in the first dimension, then we would change the first dimension of <span class="math-container">$x$</span> to be <span class="math-container">$\hat{y}$</span>.</p>
  <p>We then train the network using the <a href="https://en.wikipedia.org/wiki/Mean_squared_error#Loss_function" rel="nofollow noreferrer">Mean Squared Error Loss</a> between <span class="math-container">$x$</span> and <span class="math-container">$\tilde{x}$</span> - note that no gradient information is retained when we do the forward pass to get <span class="math-container">$\tilde{x}$</span>; in fact we usually don't use a current version of the <span class="math-container">$Q$</span> network, we use an 'old' version of the network called the target network (I imagine there's probably a question about this network on the site already so I won't explain it in detail).</p>
  <p>In case 2) the idea is much more simple as the outputs of the network are scalars so you can just train your network using the MSE between the scalar values of <span class="math-container">$Q(s, a)$</span> and <span class="math-container">$\hat{y}$</span> as defined above, the caveat here is that to calculate <span class="math-container">$\arg\max_a Q(s', a)$</span> you have to make <span class="math-container">$|\mathcal{A}|$</span> forward passes of the network for all possible <span class="math-container">$(s', a)$</span>, for fixed <span class="math-container">$s'$</span>, tuples which is why method 1) is usually preferred.</p>
  <p>Now, whilst we may use techniques from supervised learning to optimise the Q-function, you can hopefully see the differences between supervised and reinforcement learning.</p>
  <p>In short, the temporal difference is the target that you train your network towards being able to predict, and the state is (typically) the input into your neural network. For a more in depth discussion of RL being framed as supervised learning please see <a href="https://ai.stackexchange.com/questions/14167/can-supervised-learning-be-recast-as-reinforcement-learning-problem">this answer</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T12:08:05.420Z
Answers.LastActivityDate: 2021-01-18T15:47:14.193Z
Answers.LastEditDate: 2021-01-18T15:47:14.193Z
Answers.UserDisplayName: David
Body: <p>I just finished Andrew Ngs's deep learning specialization, but RL was not covered, so I don't know the basics of RL. So, I have been having trouble understanding the cost function in deep Q-learning. Like other cost functions in machine learning, you usually have <span class="math-container">$\hat{y}$</span> (the network prediction) and <span class="math-container">$y$</span> (the target, or what the network is being optimized for.)</p>
  <p>I've read through a few online articles on deep Q-learning. So far, there has been no mention of setting up a target state (<span class="math-container">$y$</span>) for the agent to produce. There has been mention of calculating a temporal-difference, however, which is where I am confused.</p>
  <p>When calculating the cost function, are you taking the input state (<span class="math-container">$\hat{y}$</span>) and a target state (<span class="math-container">$y$</span>) into consideration to determine the temporal-difference?</p>
  <p>Otherwise, I'm not sure how the cost function could determine a reward based on the input alone (state of the environment the agent is in.).</p>
  
Tags: <reinforcement-learning><q-learning><dqn><objective-functions><temporal-difference-methods>
Title: When calculating the cost in deep Q-learning, do we use both the input and target states?
UserDisplayName: junfanbl
ThreadText: # When calculating the cost in deep Q-learning, do we use both the input and target states?
  
  (Asked by: username_0 on 2021-01-17)
  
  I just finished Andrew Ngs's deep learning specialization, but RL was not covered, so I don't know the basics of RL. So, I have been having trouble understanding the cost function in deep Q-learning. Like other cost functions in machine learning, you usually have $\hat{y}$ (the network prediction) and $y$ (the target, or what the network is being optimized for.)
  
  I've read through a few online articles on deep Q-learning. So far, there has been no mention of setting up a target state ($y$) for the agent to produce. There has been mention of calculating a temporal-difference, however, which is where I am confused.
  
  When calculating the cost function, are you taking the input state ($\hat{y}$) and a target state ($y$) into consideration to determine the temporal-difference?
  
  Otherwise, I'm not sure how the cost function could determine a reward based on the input alone (state of the environment the agent is in.).
  
  **Tags:** reinforcement-learning, q-learning, dqn, objective-functions, temporal-difference-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-18. Score: 3 | Accepted answer
  
  I will first explain *briefly* to you the difference between supervised learning and reinforcement learning to make sure that you don't have any misunderstandings. In supervised learning you are provided with some data $\{(\textbf{x}\_i, y\_i)\}\_{i=1}^n$ where $\textbf{x}\_i$ are the features for data point $i$ and $y\_i$ is its true label. Now, the aim of supervised learning is to learn a function $f$ that can accurately predict the label of a data point given its features. In deep learning this function is a neural network, $f\_\theta(\cdot)$. To optimise the parameters we obtain the models prediction for the label of $y$, denoted typically by $\hat{y} = f\_\theta(x)$ and we look to optimise the parameters of the function $\theta$ by minimising the loss $\mathcal{L}(\hat{y}, y)$ (note that here the loss is a function $\mathcal{L}: \mathcal{Y} \times \mathcal{Y} \rightarrow \mathbb{R}$.
  
  In reinforcement learning things are quite different. We are not provided with any data. Instead we have a [Markov Decision Process (MDP)](https://en.wikipedia.org/wiki/Markov_decision_process) and an environment that we can interact with. The state space is defined by the MDP and we can get samples, mainly tuples of the form $(s, a, r, s')$, that we can use to teach our agent how to find an optimal policy - typically an optimal policy is one that takes an action given the current state that will maximise the sum of the future rewards of the episode. So in reinforcement you can probably see that things are quite different to supervised learning, mainly in the way that we 'obtain' our data.
  
  Of course these explanations are gross oversimplifications of the learning process in both paradigms and should only be used as a brief example to try to emphasise the different between the two learning paradigms.
  
  Now, there are two ways you can parameterise your Q-function using a neural network:
  
  1. The network takes as input the current state and outputs Q-values for each potential action; i.e. the output is a vector in $\mathbb{R}^{|\mathcal{A}|}$;
  2. The network takes as input the state and action and outputs a real number which is the Q-value for the state, action tuple you pass as input to the network.
  
  The second way is usually reserved for instances where you have a huge action space but only ever would consider a few feasible actions at each state - this saves computational complexity.
  
  Now, to answer your question:
  
  In case 1) you are assumed to have access to a transition tuple $(s, a, r, s')$. The temporal difference, which we will use as the target, is $\hat{y} = r + \max\_a Q(s', a)$. Now, as our output of $Q(s', \cdot)$ is an $\mathbb{R}^{|\mathcal{A}|}$ what we do is make a forward pass of the network for the current $(s, a)$ tuple, i.e. we get $x = Q(s, a)$, and then change the element that corresponds to the action which satisfies $\arg\max\_a Q(s', a)$ of $x$ to $\hat{y}$, so we have now got our augmented input $\tilde{x}$ which serves as our target.
  
  To make that step a bit clearer, suppose we have a 2-dimensional action space and the $\arg\max\_a Q(s', a)$ is the action in the first dimension, then we would change the first dimension of $x$ to be $\hat{y}$.
  
  We then train the network using the [Mean Squared Error Loss](https://en.wikipedia.org/wiki/Mean_squared_error#Loss_function) between $x$ and $\tilde{x}$ - note that no gradient information is retained when we do the forward pass to get $\tilde{x}$; in fact we usually don't use a current version of the $Q$ network, we use an 'old' version of the network called the target network (I imagine there's probably a question about this network on the site already so I won't explain it in detail).
  
  In case 2) the idea is much more simple as the outputs of the network are scalars so you can just train your network using the MSE between the scalar values of $Q(s, a)$ and $\hat{y}$ as defined above, the caveat here is that to calculate $\arg\max\_a Q(s', a)$ you have to make $|\mathcal{A}|$ forward passes of the network for all possible $(s', a)$, for fixed $s'$, tuples which is why method 1) is usually preferred.
  
  Now, whilst we may use techniques from supervised learning to optimise the Q-function, you can hopefully see the differences between supervised and reinforcement learning.
  
  In short, the temporal difference is the target that you train your network towards being able to predict, and the state is (typically) the input into your neural network. For a more in depth discussion of RL being framed as supervised learning please see [this answer](https://ai.stackexchange.com/questions/14167/can-supervised-learning-be-recast-as-reinforcement-learning-problem).
  
  
  ---
  
  