Answers.Body: <p>A (stochastic) <strong>policy</strong> is a set of conditional probability distributions, <span class="math-container">$$\pi(a \mid S=s), \forall s \in \mathcal{S}.$$</span> If the policy is deterministic, then it is a function <span class="math-container">$$\pi: \mathcal{S} \rightarrow \mathcal{A},$$</span> so <span class="math-container">$\pi(s) = a$</span> is the action that policy <span class="math-container">$a$</span> returns in the state <span class="math-container">$s$</span> - it always produces this same action for a given state, unless it's a <a href="https://ai.stackexchange.com/q/13088/2444">non-stationary policy</a>. A policy is also called a <em>strategy</em> (in game theory). To be usable, a stochastic policy must be turned into a <a href="https://en.wikipedia.org/wiki/Decision_rule" rel="nofollow noreferrer">decision rule</a>, i.e. you need to sample from it. A stochastic policy generalises a deterministic one.</p>
  <p>The <strong>rewards</strong> are the outputs of the <strong>reward function</strong>. A reward function can be deterministic (which is often the case) or stochastic. It can be defined as <span class="math-container">$$R : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{R},$$</span> where <span class="math-container">$\mathcal{R} \in \mathbb{R}$</span> is the reward space. If it's stochastic, then</p>
  <p><span class="math-container">\begin{align}
  R(s, a) 
  &amp;= \mathbb{E}\left[ R_t \mid S_t = s, A_t = a\right] \\
  &amp;=\sum_r r p(r \mid S_t = a, A_t=a),
  \end{align}</span>
  where <span class="math-container">$R_t$</span> is the random variable that represents the reward at time step <span class="math-container">$t$</span> and assuming a finite MDP. Stochastic reward functions generalise deterministic ones.</p>
  <p>So, policies are probability distributions or functions, while rewards are numbers. So, <strong>there's a difference between their definitions, even though they are related</strong>.</p>
  <p>How are they related? In different ways. The most important one is that an optimal policy for a given MDP is the one that, if followed, maximises the expected return, which is a function of the reward (typically, a discounted  sum of rewards). The definition of an optimal policy makes more sense if you also know the definition of a <strong>value function</strong> - I recommend you take <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Sutton &amp; Barto's book</a> and read the relevant sections.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:35:33.017Z
Answers.Text: Detailed answer, thanks. But I still don't get how these two things are different under the hood, meaning the neural network output.
Answers.UserDisplayName: Antonis Karvelas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:37:13.120Z
Answers.Text: In deep RL, the neural network can represent the policy. To find the optimal policy, you need to explore the environment and collect rewards and understand which actions lead to more rewards. You then use these rewards to update the neural network, so the policy. So, the rewards are used to train the neural network. Maybe you should look at the equations of Q-learning or any other RL algorithm. In them, $Q(s, a)$ is the neural network, while $r$ is the reward. There's clearly a difference. One is a function, the other is a number, as I say in my answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:40:37.923Z
Answers.Text: I recommend that you read the first chapters of the book mentioned in my answer. I think you don't understand the difference because you're missing some details, but I don't know which ones. Let me give you this example - hopefully, you're more familiar with supervised learning. Do you think that labels (e.g. dog or cat) are the same thing as a neural network that classifies dogs and cats? Here, the labels would be analogous to the rewards, while the classifier would be analogous to the policy. See [this](https://ai.stackexchange.com/a/14168/2444).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:47:08.427Z
Answers.Text: Maybe I'm confused because a lot of papers, articles etc. refer to them as distinct things. What you're telling me I think is that the policy is the method (nn or other) and the reward is how we get to the method (which could also be value+advantage or something else).
Answers.UserDisplayName: Antonis Karvelas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:50:39.663Z
Answers.Text: I wouldn't say that a policy is a method, if you're using it as a synonym for algorithm, although a decision rule can be thought of as an algorithm. Maybe you meant model (and not method)? A neural network is essentially a model, i.e. it can be used to mimic something. That something is usually a function. Any (continuous) function. That's why neural networks are called function approximators, i.e. they approximate functions. In this case, the function that the neural network approximates is the policy. A policy is the solution to an optimisation problem defined in terms of an MDP.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:51:26.420Z
Answers.Text: So, we can say that a policy is the "solution" to an MDP, which is the "problem". As part of the definition of the MPD, you have a reward function. So, a problem is defined in terms of the reward function. Typically, the optimization problem you want to solve is defined in terms of this reward function. I could probably go on and on and try to explain more things, like the value function, but this discussion will become long. That's why I recommend you read the book. RL is not as trivial as it may seem at the beginning. It requires some time to get used to all the definitions and details.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:54:38.570Z
Answers.Text: Feel free also to ask more separate questions on the site, but comments should not be used for long discussions.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T20:32:29.430Z
Answers.Text: Thank you, this cleared up many things. It's just that the bibliography sometimes is a bit confusing.
Answers.UserDisplayName: Antonis Karvelas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:22:46.263Z
Answers.LastActivityDate: 2022-04-20T17:30:53.380Z
Answers.LastEditDate: 2022-04-20T17:30:53.380Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Typically, the answer to a control problem in reinforcement learning (RL), is &quot;What is the policy that maximises total reward?&quot;.</p>
  <p>In really simple scenarios, that you might study to understand RL basics, this can be so obvious that you <em>could</em> just search ahead and discover the correct action without really using RL.</p>
  <p>However, sticking to the formal definitions in RL allows you to tackle harder problems, where it is not obvious how to decide what to do, or maybe even how to access the best rewards.</p>
  <p>In the formalism of RL:</p>
  <ul>
  <li><p>Reward is a real-valued signal received after each time step. For RL theory to be useful, the distrbution of reward values should be the same if start state and action are the same (you may also make the distribution depend on the next state, but that doesn't change the rule, it just means reward and next state will be correlated).</p>
  </li>
  <li><p>A policy is a function for action choice, it takes the state as input, and returns a <em>distribution</em> over all possible actions <span class="math-container">$\pi(a | s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbb{Pr}\{A_t=a|S_t=s\}$</span>. It may be deterministic and directly return a chosen action <span class="math-container">$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$</span></p>
  </li>
  </ul>
  <p>These are clearly very different things. Often we are interested in finding a policy that maximises an aggregate of rewards, but that doesn't make the policy simply related to rewards. Think of a chess game - which piece should you move to win? There is no immediate reward, and the consequence for a good or bad move will happen much later. The relationship between the policy in an early game state, and the reward at the end (e.g. +1 for a win, -1 for a loss) is not at all clear.</p>
  <p>Only in the very simplest of scenarios, where taking actions directly led to already-predictable rewards, could you use your idea of <span class="math-container">$\pi(s) = \text{argmax}_a r(s, a)$</span> where <span class="math-container">$r(s,a)$</span> was a reward function. For a start, this will not help you look ahead more than one time step. What if the best reward now was followed by a really bad reward?</p>
  <p>RL has the toolkit you can use to decide how to offset immediate vs future gains, and also how to learn what to do when you don't already have a simple function <span class="math-container">$r(s,a)$</span> which tells you what is going to happen in advance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-04-20T17:10:28.890Z
Answers.LastActivityDate: 2022-04-20T17:10:28.890Z
Answers.UserDisplayName: Neil Slater
Body: <p>I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the maximum reward)? What is different about the policy? An extra softmax applied?</p>
  
Tags: <reinforcement-learning><comparison><rewards><policies>
Title: What is the difference between a policy and rewards?
UserDisplayName: Antonis Karvelas
ThreadText: # What is the difference between a policy and rewards?
  
  (Asked by: username_0 on 2022-04-20)
  
  I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the maximum reward)? What is different about the policy? An extra softmax applied?
  
  **Tags:** reinforcement-learning, comparison, rewards, policies
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-04-20. Score: 1 | Accepted answer
  
  A (stochastic) **policy** is a set of conditional probability distributions, $$\pi(a \mid S=s), \forall s \in \mathcal{S}.$$ If the policy is deterministic, then it is a function $$\pi: \mathcal{S} \rightarrow \mathcal{A},$$ so $\pi(s) = a$ is the action that policy $a$ returns in the state $s$ - it always produces this same action for a given state, unless it's a [non-stationary policy](https://ai.stackexchange.com/q/13088/2444). A policy is also called a *strategy* (in game theory). To be usable, a stochastic policy must be turned into a [decision rule](https://en.wikipedia.org/wiki/Decision_rule), i.e. you need to sample from it. A stochastic policy generalises a deterministic one.
  
  The **rewards** are the outputs of the **reward function**. A reward function can be deterministic (which is often the case) or stochastic. It can be defined as $$R : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{R},$$ where $\mathcal{R} \in \mathbb{R}$ is the reward space. If it's stochastic, then
  
  \begin{align}
  R(s, a)
  &= \mathbb{E}\left[ R\_t \mid S\_t = s, A\_t = a\right] \\
  &=\sum\_r r p(r \mid S\_t = a, A\_t=a),
  \end{align}
  where $R\_t$ is the random variable that represents the reward at time step $t$ and assuming a finite MDP. Stochastic reward functions generalise deterministic ones.
  
  So, policies are probability distributions or functions, while rewards are numbers. So, **there's a difference between their definitions, even though they are related**.
  
  How are they related? In different ways. The most important one is that an optimal policy for a given MDP is the one that, if followed, maximises the expected return, which is a function of the reward (typically, a discounted sum of rewards). The definition of an optimal policy makes more sense if you also know the definition of a **value function** - I recommend you take [Sutton & Barto's book](http://incompleteideas.net/book/RLbook2020.pdf) and read the relevant sections.
  
  > **Comments:**
  > **username_0**: Detailed answer, thanks. But I still don't get how these two things are different under the hood, meaning the neural network output.  
  > **username_1**: In deep RL, the neural network can represent the policy. To find the optimal policy, you need to explore the environment and collect rewards and understand which actions lead to more rewards. You then use these rewards to update the neural network, so the policy. So, the rewards are used to train the neural network. Maybe you should look at the equations of Q-learning or any other RL algorithm. In them, $Q(s, a)$ is the neural network, while $r$ is the reward. There's clearly a difference. One is a function, the other is a number, as I say in my answer.  
  > **username_1**: I recommend that you read the first chapters of the book mentioned in my answer. I think you don't understand the difference because you're missing some details, but I don't know which ones. Let me give you this example - hopefully, you're more familiar with supervised learning. Do you think that labels (e.g. dog or cat) are the same thing as a neural network that classifies dogs and cats? Here, the labels would be analogous to the rewards, while the classifier would be analogous to the policy. See [this](https://ai.stackexchange.com/a/14168/2444).  
  > **username_0**: Maybe I'm confused because a lot of papers, articles etc. refer to them as distinct things. What you're telling me I think is that the policy is the method (nn or other) and the reward is how we get to the method (which could also be value+advantage or something else).  
  > **username_1**: I wouldn't say that a policy is a method, if you're using it as a synonym for algorithm, although a decision rule can be thought of as an algorithm. Maybe you meant model (and not method)? A neural network is essentially a model, i.e. it can be used to mimic something. That something is usually a function. Any (continuous) function. That's why neural networks are called function approximators, i.e. they approximate functions. In this case, the function that the neural network approximates is the policy. A policy is the solution to an optimisation problem defined in terms of an MDP.  
  > **username_1**: So, we can say that a policy is the "solution" to an MDP, which is the "problem". As part of the definition of the MPD, you have a reward function. So, a problem is defined in terms of the reward function. Typically, the optimization problem you want to solve is defined in terms of this reward function. I could probably go on and on and try to explain more things, like the value function, but this discussion will become long. That's why I recommend you read the book. RL is not as trivial as it may seem at the beginning. It requires some time to get used to all the definitions and details.  
  > **username_1**: Feel free also to ask more separate questions on the site, but comments should not be used for long discussions.  
  > **username_0**: Thank you, this cleared up many things. It's just that the bibliography sometimes is a bit confusing.  
  
  ---
  
  ### Answer by username_2 on 2022-04-20. Score: 0
  
  Typically, the answer to a control problem in reinforcement learning (RL), is "What is the policy that maximises total reward?".
  
  In really simple scenarios, that you might study to understand RL basics, this can be so obvious that you *could* just search ahead and discover the correct action without really using RL.
  
  However, sticking to the formal definitions in RL allows you to tackle harder problems, where it is not obvious how to decide what to do, or maybe even how to access the best rewards.
  
  In the formalism of RL:
  
  * Reward is a real-valued signal received after each time step. For RL theory to be useful, the distrbution of reward values should be the same if start state and action are the same (you may also make the distribution depend on the next state, but that doesn't change the rule, it just means reward and next state will be correlated).
  * A policy is a function for action choice, it takes the state as input, and returns a *distribution* over all possible actions $\pi(a | s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbb{Pr}\{A\_t=a|S\_t=s\}$. It may be deterministic and directly return a chosen action $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$
  
  These are clearly very different things. Often we are interested in finding a policy that maximises an aggregate of rewards, but that doesn't make the policy simply related to rewards. Think of a chess game - which piece should you move to win? There is no immediate reward, and the consequence for a good or bad move will happen much later. The relationship between the policy in an early game state, and the reward at the end (e.g. +1 for a win, -1 for a loss) is not at all clear.
  
  Only in the very simplest of scenarios, where taking actions directly led to already-predictable rewards, could you use your idea of $\pi(s) = \text{argmax}\_a r(s, a)$ where $r(s,a)$ was a reward function. For a start, this will not help you look ahead more than one time step. What if the best reward now was followed by a really bad reward?
  
  RL has the toolkit you can use to decide how to offset immediate vs future gains, and also how to learn what to do when you don't already have a simple function $r(s,a)$ which tells you what is going to happen in advance.
  
  
  ---
  
  