Answers.Body: <p>Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly.</p>
  <p>In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs <span class="math-container">$x_i, y_i$</span> where <span class="math-container">$x_i$</span> represents a state and <span class="math-container">$y_i$</span> represents the correct action to take in that state in order to maximise the expected return. For simplicity I will use <span class="math-container">$G_t = \sum_{k=1}^{\infty} \gamma^k R_{t+k+1}$</span> for return here (where <span class="math-container">$G_t$</span> and <span class="math-container">$R_t$</span> are random variables), there are other definitions, but the argument that follows doesn't change much for them.</p>
  <p>You can use the oracle to reduce the RL training process to SL, creating a policy function <span class="math-container">$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$</span> learned from a dataset that the oracle output. This clearly relates SL with RL, but how do <span class="math-container">$x_i, y_i$</span> from SL relate to <span class="math-container">$s_t, a_t$</span> from RL in terms of reward values?</p>
  <p>The states can relate directly (as input):</p>
  <p><span class="math-container">$$x_i \equiv s_t$$</span></p>
  <p>The action from the policy function is more indirect, if you want to see how reward is involved:</p>
  <p><span class="math-container">$$y_i \equiv \pi^*(s_t) = \text{argmax}_a \mathbb{E}_{A \sim \pi^*}[\sum_{k=1}^{\infty} \gamma^k R_{t+k+1} | S_t=s_t, A_t=a]$$</span></p>
  <p>Note the oracle is represented by the optimal policy function <span class="math-container">$\pi^*(s_t)$</span>, and the expectation is conditional both on the start conditions of state and action plus following the optimal policy from then on (which is what <span class="math-container">$A \sim \pi^*$</span> is representing).</p>
  <p>In practice the optimal policy function is unknown when starting RL, so the learning process <em>cannot</em> be reduced to a SL problem. However, you can get close in some circumstances by creating a dataset of action choices made by an expert at the problem. In that case a similar relationship applies - the label (of which action to take) and immediate reward are different things but can be related by noting that the expert behaviour is close to the <span class="math-container">$\text{argmax}$</span> over actions of expected sums of future reward.</p>
  <p>Another way to view the difference:</p>
  <ul>
  <li><p>In SL, the signal from the label is an <em>instruction</em> - &quot;associate these two values&quot;. Data is supplied to the learning process by some other independent process, and can be learned from directly</p>
  </li>
  <li><p>In RL, the signal from the reward is a <em>consequence</em> - &quot;this is the value, in context, of what you just did&quot;, and needs to be learned from indirectly. Data is not supplied separately from the learning process, but must be actively collected by it - deciding which state, action pairs to learn from is part of the agent's learning task</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T09:45:07.903Z
Answers.Text: Excellent answer. Exactly what I was looking for. Indeed, this _indirect dependency_ (which I was missing) is what differentiates them. On a side note, from what I've read the reward distribution _can_ be noisy. Is it possible in supervised learning that the labels themselves are less reliable? We usually take it for granted that the labels are produced by some almighty oracle.
Answers.UserDisplayName: Saptam
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T10:27:50.843Z
Answers.Text: @Saptam: Yes, supervised learning works with noisy labels, and that is a common assumption. Most SL methods will do the correct thing of learning an expected value based on statistics of the labels in the dataset.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-08T08:29:18.687Z
Answers.LastActivityDate: 2020-07-08T10:28:31.437Z
Answers.LastEditDate: 2020-07-08T10:28:31.437Z
Answers.UserDisplayName: Neil Slater
Body: <p>How is the notion of immediate <em>reward</em> used in the reinforcement learning different from the notion of a <em>label</em> we find in the supervised learning problems?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-07T15:17:55.447Z
Comments.Text: Does this answer your question? [Can supervised learning be recast as reinforcement learning problem?](https://ai.stackexchange.com/questions/14167/can-supervised-learning-be-recast-as-reinforcement-learning-problem) This is not an exact duplicate, but your question is very related to the other one. If you think the answer and question in the other link are useful, feel free to upvote both of them.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-07T15:40:26.197Z
Comments.Text: @nbro Indeed both the question and the answer that you gave is very interesting (sadly I don't have enough reputation to upvote them right now). But my question is slightly different. I'm not seeking to _convert_ one type of problem to another. My intention is to know how much do they resemble and contrast each other.
Comments.UserDisplayName: Saptam
Tags: <reinforcement-learning><comparison><rewards><supervised-learning>
Title: How is the reward in reinforcement learning different from the label in supervised learning problems?
UserDisplayName: Saptam
ThreadText: # How is the reward in reinforcement learning different from the label in supervised learning problems?
  
  (Asked by: username_0 on 2020-07-07)
  
  How is the notion of immediate *reward* used in the reinforcement learning different from the notion of a *label* we find in the supervised learning problems?
  
  **Tags:** reinforcement-learning, comparison, rewards, supervised-learning
  > **Comments:**
  > **username_1**: Does this answer your question? [Can supervised learning be recast as reinforcement learning problem?](https://ai.stackexchange.com/questions/14167/can-supervised-learning-be-recast-as-reinforcement-learning-problem) This is not an exact duplicate, but your question is very related to the other one. If you think the answer and question in the other link are useful, feel free to upvote both of them.  
  > **username_0**: @nbro Indeed both the question and the answer that you gave is very interesting (sadly I don't have enough reputation to upvote them right now). But my question is slightly different. I'm not seeking to _convert_ one type of problem to another. My intention is to know how much do they resemble and contrast each other.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-08. Score: 3 | Accepted answer
  
  Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly.
  
  In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs $x\_i, y\_i$ where $x\_i$ represents a state and $y\_i$ represents the correct action to take in that state in order to maximise the expected return. For simplicity I will use $G\_t = \sum\_{k=1}^{\infty} \gamma^k R\_{t+k+1}$ for return here (where $G\_t$ and $R\_t$ are random variables), there are other definitions, but the argument that follows doesn't change much for them.
  
  You can use the oracle to reduce the RL training process to SL, creating a policy function $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ learned from a dataset that the oracle output. This clearly relates SL with RL, but how do $x\_i, y\_i$ from SL relate to $s\_t, a\_t$ from RL in terms of reward values?
  
  The states can relate directly (as input):
  
  $$x\_i \equiv s\_t$$
  
  The action from the policy function is more indirect, if you want to see how reward is involved:
  
  $$y\_i \equiv \pi^\*(s\_t) = \text{argmax}\_a \mathbb{E}\_{A \sim \pi^\*}[\sum\_{k=1}^{\infty} \gamma^k R\_{t+k+1} | S\_t=s\_t, A\_t=a]$$
  
  Note the oracle is represented by the optimal policy function $\pi^\*(s\_t)$, and the expectation is conditional both on the start conditions of state and action plus following the optimal policy from then on (which is what $A \sim \pi^\*$ is representing).
  
  In practice the optimal policy function is unknown when starting RL, so the learning process *cannot* be reduced to a SL problem. However, you can get close in some circumstances by creating a dataset of action choices made by an expert at the problem. In that case a similar relationship applies - the label (of which action to take) and immediate reward are different things but can be related by noting that the expert behaviour is close to the $\text{argmax}$ over actions of expected sums of future reward.
  
  Another way to view the difference:
  
  * In SL, the signal from the label is an *instruction* - "associate these two values". Data is supplied to the learning process by some other independent process, and can be learned from directly
  * In RL, the signal from the reward is a *consequence* - "this is the value, in context, of what you just did", and needs to be learned from indirectly. Data is not supplied separately from the learning process, but must be actively collected by it - deciding which state, action pairs to learn from is part of the agent's learning task
  
  > **Comments:**
  > **username_0**: Excellent answer. Exactly what I was looking for. Indeed, this _indirect dependency_ (which I was missing) is what differentiates them. On a side note, from what I've read the reward distribution _can_ be noisy. Is it possible in supervised learning that the labels themselves are less reliable? We usually take it for granted that the labels are produced by some almighty oracle.  
  > **username_2**: @Saptam: Yes, supervised learning works with noisy labels, and that is a common assumption. Most SL methods will do the correct thing of learning an expected value based on statistics of the labels in the dataset.  
  
  ---
  
  