Answers.Body: <p>Ultimately, in RL, the policy is what you want to find. It's the solution to the Markov Decision Process (MDP). But you don't want to find any policy, but the <em>optimal policy</em>, i.e. the one that will make the agent collect the highest amount of reward in the long run (i.e. the highest <em>return</em>), if followed.</p>
  <p>In deep RL, the policy might be represented by a neural network, which gets a state as input and produces a probability distribution over actions, which we can be denoted by <span class="math-container">$\pi(a \mid s; \theta)$</span>, where <span class="math-container">$\theta$</span> is the parameter vector. If you change <span class="math-container">$\theta$</span>, you also change the output of the policy.</p>
  <p>The reward function function tells how good the actions that the agent takes are. So, it can be defined as the function <span class="math-container">$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{R}$</span>, where <span class="math-container">$\mathcal{R} \subset \mathbb{R}$</span> is the reward space. So, <span class="math-container">$r(s, a)$</span> is the reward that the agent receives for taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span>. For example, in the game of chess, if you win the game, <span class="math-container">$r$</span> could return <span class="math-container">$1$</span>, while, if you lose the game, it could return <span class="math-container">$-1$</span>. The reward function is usually pre-defined, in the sense that it's part of the problem definition. You don't have to learn it, although you can learn reward functions with <em>inverse RL</em> techniques.</p>
  <p>So, the reward function is not the objective/loss function, but the objective function is usually defined in terms of the reward function, in the same way that the mean squared error (MSE) in supervised learning is defined in terms of the correct labels or targets.</p>
  <p>Now, what could be the loss function in RL? It depends on how you train the RL agent. For example, in <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">DQN</a>, the loss function is</p>
  <p><span class="math-container">$$
  L_{i}\left(\theta_{i}\right)=\mathbb{E}_{s, a \sim \rho(\cdot)}\left[\left(y_{i}-Q\left(s, a ; \theta_{i}\right)\right)^{2}\right]
  $$</span>
  where</p>
  <p><span class="math-container">$$
  y_{i}=\mathbb{E}_{s^{\prime} \sim \mathcal{E}}\left[\underbrace{r}_{\text{Reward}}+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta_{i-1}\right) \mid s, a\right]
  $$</span>
  is the target value for <span class="math-container">$Q\left(s, a ; \theta_{i}\right)$</span>, which is what we're trying to learn and it's represented by the neural network with parameters <span class="math-container">$\theta_{i}$</span>. <span class="math-container">$Q$</span> is known as the <em>value function</em>, which is defined as the expected return, from which we can derive the policy. So, it gets as input a state and an action, not the reward. It produces an estimate of the expected return, which is defined as the sum of rewards.</p>
  <p>This answer should answer all your questions and doubts. See also <a href="https://ai.stackexchange.com/a/14168/2444">this answer</a> about the relationship between supervised learning and reinforcement learning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T13:18:52.330Z
Answers.LastActivityDate: 2022-03-30T20:23:39.743Z
Answers.LastEditDate: 2022-03-30T20:23:39.743Z
Answers.UserDisplayName: nbro
Answers.Body: <p><strong>1. Question: The difference between loss and reward/penalty</strong></p>
  <blockquote>
  <p>So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to learn. Am I right?</p>
  </blockquote>
  <p>You are partially right:
  You could interpret the negative reward as a <em>loss</em> that you want to minimize. But the model cannot learn from the reward directly. The reason for that is that you (usually) cannot formulate the reward as a differentiable function of the model parameters. Hence, you cannot compute a gradient purely from the reward. You need a second function - the loss - that is</p>
  <ol>
  <li>Differentiable and</li>
  <li>Depends on the parameters inside your model.</li>
  </ol>
  <p>Only then you can compute a gradient w.r.t. the model parameters and make the model learn. This already answers another question:</p>
  <blockquote>
  <p>Is it the policy related the loss function somehow?</p>
  </blockquote>
  <p>Yes! The policy HAS to be a part of the loss function, otherwise you wouldn't be able to do gradient decent to optimize the model.</p>
  <p>A simple way to get from reward to the loss is implemented in the <strong>REINFORCE</strong> algorithm. To understand its loss term, you have to know that the model does not output definitive actions, but rather a probability distribution over all possible actions. Here is the loss function that REINFORCE uses to optimize the model:</p>
  <p><span class="math-container">$loss = -log\_likelihood(action) \cdot return$</span></p>
  <p>As you can see, the loss is the product of the negative log likelihood of the action and the <span class="math-container">$return$</span>. The return correlates with the reward (Return is the discounted reward which distributes the reward received at timestep <span class="math-container">$t$</span> backwards to also reward actions that led to the reward). Intuitively this means that for a large reward, the model wants to be very certain about which action to take. So there you have it: The reward reflects how successful you are in the environment and the loss is the optimization objective maximizing the probability to take <em>good</em> actions.</p>
  <p><strong>2. Question: The Schematic doesn't include the loss function</strong></p>
  <p>The image you posted depicts how you collect the data which you use to optimize the model. You would run this loop of taking an action and receiving a reward until you have a full batch of data. On this batch you would then compute the loss and update the model. It's quite important for reinforcement learning to gather batches and not use single steps for optimization, because otherwise the resulting gradient would be very noisy and in most cases prevent proper optimization.</p>
  <p>The main issue why your gradient would be noisy is the <strong>credit assignment problem</strong>:</p>
  <p>Lets assume the environment is a grid world and the task is to walk forward for 7 steps. You start at <span class="math-container">$S$</span> and you will get a reward as soon as you reach location <span class="math-container">$G$</span>:</p>
  <p><img src="https://www.researchgate.net/profile/Markus-Dumke/publication/320890681/figure/fig1/AS:763210537922560@1558974980641/The-windy-gridworld-task-The-goal-is-to-move-from-the-start-state-S-to-the-goal-state-G.jpg" alt="gridworld" /></p>
  <p>The reward will show you that you have done <em>something</em> right but multiple actions where responsible for getting the reward (not just the last step forward). However, you never exactly know which actions where the <em>right</em> actions and which actions where actually bad. You might have taken a very inefficient route to the goal.</p>
  <p>The problem that you don't know which actions contributed to getting the reward is called the <em>credit assignment problem</em>. And in fact you can only have a good heuristic to assign the reward. This has to be compensated by computing the model update on batches rather than single steps. One such heuristic is the <a href="https://arxiv.org/abs/1506.02438" rel="noreferrer">general advantage estimate</a>. This is a function that you apply to your reward before plugging it into the loss function.</p>
  <blockquote>
  <p>Maybe my confusion has to do with identifying NN with supervised learning</p>
  </blockquote>
  <p>One major difference of supervised learning and reinforcement learning lies in the <em>credit assignment problem</em>: In supervised learning you input a sample and you know what should come out. In RL you only have a rough estimate on how good you where but you will (usually) never know what should come out of your model, because there are multiple possible ways to reach the goal.</p>
  <p>Hope this helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T14:16:29.920Z
Answers.Text: This all looks correct, but the focus on REINFORCE is adding a bit much IMO when the OP is considering the (IMO) conceptually simpler DQN. You also have the loss function wrong at the moment (it should be *return*, not reward used as multipler for log loss - you might use reward in a gradient bandit).
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-31T11:35:58.750Z
Answers.Text: Thanks for the feedback, I understand your points. You are right, I was a bit lazy putting reward in the formula and I will update this in my answer. I agree that the core concept of Q-Learning is more straight forward, although I find the loss of REINFORCE a bit easier to understand that's why I went with that. IMO it's also a bit more aligned with the RL concepts, but to be honest it's what I have been in contact with more than Q-learning - so I might be biased
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-30T12:40:01.353Z
Answers.LastActivityDate: 2022-03-31T11:39:18.037Z
Answers.LastEditDate: 2022-03-31T11:39:18.037Z
Answers.UserDisplayName: Chillston
Body: <p>In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a <em>Loss function</em>, a <em>reward/penalty</em> and the integration of both in DRL.</p>
  <ul>
  <li><p>Loss function: Given an output of the model and the ground truth, it
  measures &quot;how good&quot; the output has been. And using it, the parameters
  of the model are adjusted. For instance, MAE. But if you were working
  in Computer Vision quality, you could use, for instance, SSIM.</p>
  </li>
  <li><p>Reward: Given an agent (a model) and an environment, once the agent
  performs an action, the environment gives it a reward (or a penalty)
  to measure &quot;how good&quot; the action has been. Very simple rewards are +1
  or -1.</p>
  </li>
  </ul>
  <p>So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to learn. Am I right?</p>
  <p>Now, as for DRL. I see the typical diagram where the agent is modelled using a Neural Network (NN).</p>
  <p><a href="https://i.sstatic.net/CS6bI.png" rel="noreferrer"><img src="https://i.sstatic.net/CS6bI.png" alt="enter image description here" /></a></p>
  <p>I am trying to interpret it, but I do not understand it.</p>
  <p>Is it the policy related the loss function somehow? Where is the loss function? How does the reward feed the NN? Is it a parameter for the loss function?</p>
  <p>Maybe my confusion has to do with identifying NN with supervised learning, or with not getting this with Q-learning or so.. Can anyone help?</p>
  
Tags: <reinforcement-learning><comparison><deep-rl><objective-functions><rewards>
Title: What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?
UserDisplayName: Theo Deep
ThreadText: # What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?
  
  (Asked by: username_0 on 2022-03-30)
  
  In Deep Reinforcement Learning (DRL) I am having difficulties in understanding the difference between a *Loss function*, a *reward/penalty* and the integration of both in DRL.
  
  * Loss function: Given an output of the model and the ground truth, it
    measures "how good" the output has been. And using it, the parameters
    of the model are adjusted. For instance, MAE. But if you were working
    in Computer Vision quality, you could use, for instance, SSIM.
  * Reward: Given an agent (a model) and an environment, once the agent
    performs an action, the environment gives it a reward (or a penalty)
    to measure "how good" the action has been. Very simple rewards are +1
    or -1.
  
  So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to learn. Am I right?
  
  Now, as for DRL. I see the typical diagram where the agent is modelled using a Neural Network (NN).
  
  [![enter image description here](https://i.sstatic.net/CS6bI.png)](https://i.sstatic.net/CS6bI.png)
  
  I am trying to interpret it, but I do not understand it.
  
  Is it the policy related the loss function somehow? Where is the loss function? How does the reward feed the NN? Is it a parameter for the loss function?
  
  Maybe my confusion has to do with identifying NN with supervised learning, or with not getting this with Q-learning or so.. Can anyone help?
  
  **Tags:** reinforcement-learning, comparison, deep-rl, objective-functions, rewards
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-03-30. Score: 7 | Accepted answer
  
  **1. Question: The difference between loss and reward/penalty**
  
  > So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to learn. Am I right?
  
  You are partially right:
  You could interpret the negative reward as a *loss* that you want to minimize. But the model cannot learn from the reward directly. The reason for that is that you (usually) cannot formulate the reward as a differentiable function of the model parameters. Hence, you cannot compute a gradient purely from the reward. You need a second function - the loss - that is
  
  1. Differentiable and
  2. Depends on the parameters inside your model.
  
  Only then you can compute a gradient w.r.t. the model parameters and make the model learn. This already answers another question:
  
  > Is it the policy related the loss function somehow?
  
  Yes! The policy HAS to be a part of the loss function, otherwise you wouldn't be able to do gradient decent to optimize the model.
  
  A simple way to get from reward to the loss is implemented in the **REINFORCE** algorithm. To understand its loss term, you have to know that the model does not output definitive actions, but rather a probability distribution over all possible actions. Here is the loss function that REINFORCE uses to optimize the model:
  
  $loss = -log\\_likelihood(action) \cdot return$
  
  As you can see, the loss is the product of the negative log likelihood of the action and the $return$. The return correlates with the reward (Return is the discounted reward which distributes the reward received at timestep $t$ backwards to also reward actions that led to the reward). Intuitively this means that for a large reward, the model wants to be very certain about which action to take. So there you have it: The reward reflects how successful you are in the environment and the loss is the optimization objective maximizing the probability to take *good* actions.
  
  **2. Question: The Schematic doesn't include the loss function**
  
  The image you posted depicts how you collect the data which you use to optimize the model. You would run this loop of taking an action and receiving a reward until you have a full batch of data. On this batch you would then compute the loss and update the model. It's quite important for reinforcement learning to gather batches and not use single steps for optimization, because otherwise the resulting gradient would be very noisy and in most cases prevent proper optimization.
  
  The main issue why your gradient would be noisy is the **credit assignment problem**:
  
  Lets assume the environment is a grid world and the task is to walk forward for 7 steps. You start at $S$ and you will get a reward as soon as you reach location $G$:
  
  ![gridworld](https://www.researchgate.net/profile/Markus-Dumke/publication/320890681/figure/fig1/AS:763210537922560@1558974980641/The-windy-gridworld-task-The-goal-is-to-move-from-the-start-state-S-to-the-goal-state-G.jpg)
  
  The reward will show you that you have done *something* right but multiple actions where responsible for getting the reward (not just the last step forward). However, you never exactly know which actions where the *right* actions and which actions where actually bad. You might have taken a very inefficient route to the goal.
  
  The problem that you don't know which actions contributed to getting the reward is called the *credit assignment problem*. And in fact you can only have a good heuristic to assign the reward. This has to be compensated by computing the model update on batches rather than single steps. One such heuristic is the [general advantage estimate](https://arxiv.org/abs/1506.02438). This is a function that you apply to your reward before plugging it into the loss function.
  
  > Maybe my confusion has to do with identifying NN with supervised learning
  
  One major difference of supervised learning and reinforcement learning lies in the *credit assignment problem*: In supervised learning you input a sample and you know what should come out. In RL you only have a rough estimate on how good you where but you will (usually) never know what should come out of your model, because there are multiple possible ways to reach the goal.
  
  Hope this helps.
  
  > **Comments:**
  > **username_3**: This all looks correct, but the focus on REINFORCE is adding a bit much IMO when the OP is considering the (IMO) conceptually simpler DQN. You also have the loss function wrong at the moment (it should be *return*, not reward used as multipler for log loss - you might use reward in a gradient bandit).  
  > **username_2**: Thanks for the feedback, I understand your points. You are right, I was a bit lazy putting reward in the formula and I will update this in my answer. I agree that the core concept of Q-Learning is more straight forward, although I find the loss of REINFORCE a bit easier to understand that's why I went with that. IMO it's also a bit more aligned with the RL concepts, but to be honest it's what I have been in contact with more than Q-learning - so I might be biased  
  
  ---
  
  ### Answer by username_1 on 2022-03-30. Score: 3
  
  Ultimately, in RL, the policy is what you want to find. It's the solution to the Markov Decision Process (MDP). But you don't want to find any policy, but the *optimal policy*, i.e. the one that will make the agent collect the highest amount of reward in the long run (i.e. the highest *return*), if followed.
  
  In deep RL, the policy might be represented by a neural network, which gets a state as input and produces a probability distribution over actions, which we can be denoted by $\pi(a \mid s; \theta)$, where $\theta$ is the parameter vector. If you change $\theta$, you also change the output of the policy.
  
  The reward function function tells how good the actions that the agent takes are. So, it can be defined as the function $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{R}$, where $\mathcal{R} \subset \mathbb{R}$ is the reward space. So, $r(s, a)$ is the reward that the agent receives for taking action $a$ in state $s$. For example, in the game of chess, if you win the game, $r$ could return $1$, while, if you lose the game, it could return $-1$. The reward function is usually pre-defined, in the sense that it's part of the problem definition. You don't have to learn it, although you can learn reward functions with *inverse RL* techniques.
  
  So, the reward function is not the objective/loss function, but the objective function is usually defined in terms of the reward function, in the same way that the mean squared error (MSE) in supervised learning is defined in terms of the correct labels or targets.
  
  Now, what could be the loss function in RL? It depends on how you train the RL agent. For example, in [DQN](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf), the loss function is
  
  $$
  L\_{i}\left(\theta\_{i}\right)=\mathbb{E}\_{s, a \sim \rho(\cdot)}\left[\left(y\_{i}-Q\left(s, a ; \theta\_{i}\right)\right)^{2}\right]
  $$
  where
  
  $$
  y\_{i}=\mathbb{E}\_{s^{\prime} \sim \mathcal{E}}\left[\underbrace{r}\_{\text{Reward}}+\gamma \max \_{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta\_{i-1}\right) \mid s, a\right]
  $$
  is the target value for $Q\left(s, a ; \theta\_{i}\right)$, which is what we're trying to learn and it's represented by the neural network with parameters $\theta\_{i}$. $Q$ is known as the *value function*, which is defined as the expected return, from which we can derive the policy. So, it gets as input a state and an action, not the reward. It produces an estimate of the expected return, which is defined as the sum of rewards.
  
  This answer should answer all your questions and doubts. See also [this answer](https://ai.stackexchange.com/a/14168/2444) about the relationship between supervised learning and reinforcement learning.
  
  
  ---
  
  