Answers.Body: <p>Reinforcement learning can be confusing in relation to supervised and unsupervised learning, especially for deep learning models. Here is the simple explanation make this clearer:</p>
  <h3>Supervised Learning</h3>
  <p>In the case of supervised learning, we are given a function that maps inputs to their desired outputs based on example input-output pairs. The basic setup is like this:</p>
  <ul>
  <li><strong>Dataset:</strong> There is a dataset with inputs <span class="math-container">$(x_i)$</span> and corresponding labels <span class="math-container">$(y_i)$</span>.</li>
  <li><strong>Objective:</strong> The objective is to learn a function <span class="math-container">$(f(x))$</span> which makes the most accurate possible predictions about the labels <span class="math-container">$(y_i)$</span>.</li>
  <li><strong>Loss Function:</strong> You minimize a loss function <span class="math-container">$(L(f(x), y))$</span> over the dataset to find the best <span class="math-container">$(f)$</span>.</li>
  </ul>
  <h3>Reinforcement Learning</h3>
  <p>In reinforcement learning, rather than being shown an example of what to do and being told this is the correct action for a given input, the agent has to learn from a scalar reward signal.</p>
  <ul>
  <li><strong>Actions and States:</strong> At every time step <span class="math-container">$(t)$</span> the agent is in state <span class="math-container">$(s_t)$</span>, takes action <span class="math-container">$(a_t)$</span>, and moves to a new state <span class="math-container">$(s_{t+1})$</span>.</li>
  <li><strong>Rewards:</strong> The agent receives a reward <span class="math-container">$(r_t)$</span> after taking an action.</li>
  <li><strong>Goal:</strong> The goal is to maximize the cumulative reward, called the return <span class="math-container">$(G_t = \sum_{i=t+1}^\infty R_i)$</span>.</li>
  </ul>
  <h3>The Role of Deep Learning in RL</h3>
  <p>Deep learning is commonly used in RL to approximate functions such as the policy (what action to take for a given state) and the value function (expected return from a state). But the presence of deep learning does not fundamentally change RL.</p>
  <h3>RL vs. Supervised Learning</h3>
  <ul>
  <li><strong>Feedback:</strong> In supervised learning, you give explicit labels to each input. You provide rewards in RL: feedback signals that show the agent how to learn the best actions over time.</li>
  <li><strong>Learning Process:</strong> RL learns through exploration and exploitation, which is way more dynamic than static datasets used in supervised learning.</li>
  </ul>
  <h3>Is RL Unsupervised Learning?</h3>
  <p>Probably, some will argue that RL is unsupervised since it does not depend on labeled data like supervised learning. However, RL is not purely unsupervised because of the following two factors:</p>
  <ul>
  <li><strong>Guidance:</strong> The reward function is a form of guidance, indicating in some sense what the right thing to do is.</li>
  <li><strong>Goal-oriented:</strong> The agent learns to achieve an explicit goal—maximizing reward—a kind of supervision.</li>
  </ul>
  <p>Reinforcement learning is squarely situated within this learning paradigm:</p>
  <ul>
  <li>It is not strictly supervised in that it does not use labeled input-output pairs in a straightforward way.</li>
  <li>It is not purely unsupervised because it relies on rewards, a form of feedback that orients learning.</li>
  </ul>
  <p>Therefore, RL is often considered a separate machine learning category, which can leverage deep learning techniques to accomplish its goals.</p>
  <h3>Additional Resources</h3>
  <p>To gain deeper insights, consider watching the <a href="https://www.youtube.com/watch?v=2pWv7GOvuf0&amp;t=2s" rel="nofollow noreferrer">Reinforcement Learning Course by David Silver</a>, which takes a comprehensive approach to understanding the principles and implementations of RL.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T13:06:19.247Z
Answers.LastActivityDate: 2024-07-09T14:45:36.797Z
Answers.LastEditDate: 2024-07-09T14:45:36.797Z
Answers.UserDisplayName: Faizy
Answers.Body: <h3>Supervised learning</h3>
  
  <p>The supervised learning (SL) problem is formulated as follows. </p>
  
  <p>You are given a dataset <span class="math-container">$\mathcal{D} = \{(x_i, y_i)_{i=1}^N$</span>, which is assumed to be drawn i.i.d. from an <strong>unknown</strong> joint probability distribution <span class="math-container">$p(x, y)$</span>, where <span class="math-container">$x_i$</span> represents the <span class="math-container">$i$</span>th input and <span class="math-container">$y_i$</span> is the corresponding label. You choose a loss function <span class="math-container">$\mathcal{L}: V \times U \rightarrow \mathbb{R}$</span>. Then your goal is to minimize the so-called <em>empirical risk</em> </p>
  
  <p><span class="math-container">$$R_{\mathcal{D}}[f]=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(x_i, f(x_i)) \tag{0}\label{0}$$</span></p>
  
  <p>with respect to <span class="math-container">$f$</span>. In other words, you want to find the <span class="math-container">$f$</span> that minimizes the average above, which can also be formally written as
  <span class="math-container">$$
  f^* = \operatorname{argmin}_f R[f] \tag{1}\label{1}
  $$</span>
  The problem \ref{1} is called the <strong>empirical risk minimization</strong> because it is a proxy problem for the <strong>expected risk minimization</strong> (but you can ignore this for now).</p>
  
  <h3>Reinforcement learning</h3>
  
  <p>In reinforcement learning, you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. At each time step <span class="math-container">$t$</span>, the agent takes <span class="math-container">$a_t$</span> in the state <span class="math-container">$s_t$</span>, receives a reward <span class="math-container">$r_t$</span> from the environment and the agent and the environment move to another state <span class="math-container">$s_{t+1}$</span>. </p>
  
  <p>The goal of the agent is to maximize the <strong>expected return</strong> </p>
  
  <p><span class="math-container">$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$</span> </p>
  
  <p>where <span class="math-container">$t$</span> is the current time step (so we don't care about the past), <span class="math-container">$R_i$</span> is a random variable that represents the probable reward at time step <span class="math-container">$i$</span>, and <span class="math-container">$G_t = \sum_{i=t+1}^\infty R_i $</span> is the so-called <em>return</em> (i.e. a sum of future rewards, in this case, starting from time step <span class="math-container">$t$</span>), which is also a random variable.</p>
  
  <p>In this context, the most important job of the programmer is to define a function <span class="math-container">$\mathcal{R}(s, a)$</span>, the reward function, which provides the <em>reinforcement</em> (or reward) signal to the RL agent. <span class="math-container">$\mathcal{R}(s, a)$</span> will deterministically or stochastically determine the reward that the agent receives every time it takes action <span class="math-container">$a$</span> in the state <span class="math-container">$s$</span>. (Note that <span class="math-container">$\mathcal{R}$</span> is different from <span class="math-container">$R_i$</span>, which is a random variable that represents the reward at time step <span class="math-container">$i$</span>).</p>
  
  <h3>What is the difference between SL and RL?</h3>
  
  <p>In RL, you (the programmer) need to define the reward function <span class="math-container">$\mathcal{R}$</span> and you want to maximize the expected return. On the other hand, in SL you are given (or you collect) a dataset <span class="math-container">$\mathcal{D}$</span>, you choose <span class="math-container">$\mathcal{L}$</span> in \ref{0}, and the goal is to find the function <span class="math-container">$f^*$</span> that minimizes the empirical risk.  So, these have different settings and goals, so they are different!</p>
  
  <p>However, every SL problem can be cast as an RL problem. See <a href="https://ai.stackexchange.com/a/14168/2444">this answer</a>. Similarly, in certain cases, you can formulate an RL as an SL problem. So, although the approaches are different, they are related.</p>
  
  <h3>Is RL an unsupervised learning approach?</h3>
  
  <p>In RL, you do <strong>not</strong> tell the agent what <strong>action</strong> it needs to take. You only say that the action that was taken was "bad", "good" or "so so". The agent needs to figure out which actions to take based on your feedback. In SL, you explicitly say that, for this input <span class="math-container">$x_i$</span>, the output should be <span class="math-container">$y_i$</span>.</p>
  
  <p>Some people may consider RL is an unsupervised learning approach, but I think this is wrong, because, in RL, the programmer still needs to define the reward function, so RL isn't totally unsupervised and it's also not totally supervised. For this reason, many people consider RL an approach that sits between UL and SL.</p>
  
  <h3>What is deep learning?</h3>
  
  <p>The term/expression deep learning (DL) refers to the use of deep neural networks (i.e. neural networks with many layers, where "many" can refer to more than 1 or 1000, i.e. it depends on the context) in machine learning, either supervised, unsupervised, or reinforcement learning. So, you can apply deep learning to SL, RL and UL. So, DL is not only restricted to SL.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T17:46:03.983Z
Answers.Text: Nice answer. Since the OP asked about Deep Q-Learning, it might be worth clarifying the role of DL in that algorithm, along the following lines: a popular method for solving RL problems is Q-Learning, where one learns the Q-value function. In some RL problems with large state or action spaces, this function is difficult to compute directly, so we approximate it with a DNN—this is Deep Q-Learning. So there is no essential connection between RL and DL—it's just that some value functions are too complex to learn any other way (as far as we know).
Answers.UserDisplayName: ptal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T09:30:25.843Z
Answers.Text: @ptal Your explanation of DQN is correct, but I'd not go that far to say that there's no connection between RL and DL. They are both try to come up with solutions to the general learning problem and they can be combined, so there's a connection in this sense.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T18:51:17.460Z
Answers.Text: Yes, you're right—it would be misleading to say that there is no connection between DL and RL (though I did say "no *essential* connection," by which I meant that it is possible to do RL without DL, contrary to what OP's instructor seemed to claim).
Answers.UserDisplayName: ptal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T13:06:01.620Z
Answers.LastActivityDate: 2020-05-05T14:49:37.570Z
Answers.LastEditDate: 2020-05-05T14:49:37.570Z
Answers.UserDisplayName: nbro
Body: <p>I was watching a video in my online course where I'm learning about A.I. I am a very beginner in it. </p>
  
  <p>At one point in the course, the instructor says that reinforcement learning (RL) needs a deep learning model (NN) to perform an action. But for that, we need expected results in our model for the NN to learn how to predict the Q-values. </p>
  
  <p>Nevertheless, at the beginning of the course, they said to me that RL is an unsupervised learning approach because the agent performs the action, receives the response from the environment, and finally takes the more likely action, that is, with the highest Q value. </p>
  
  <p>But if I'm using deep learning in RL, for me, RL looks like a supervised learning approach. I'm a little confused about these things, could someone give me clarifications about them?</p>
  
  <p><a href="https://i.sstatic.net/JuKV7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JuKV7.png" alt="Example from Online course"></a></p>
  
Tags: <deep-learning><reinforcement-learning><terminology><unsupervised-learning><supervised-learning>
Title: How can reinforcement learning be unsupervised learning if it uses deep learning?
UserDisplayName: Lucas Vital
ThreadText: # How can reinforcement learning be unsupervised learning if it uses deep learning?
  
  (Asked by: username_0 on 2020-05-04)
  
  I was watching a video in my online course where I'm learning about A.I. I am a very beginner in it.
  
  At one point in the course, the instructor says that reinforcement learning (RL) needs a deep learning model (NN) to perform an action. But for that, we need expected results in our model for the NN to learn how to predict the Q-values.
  
  Nevertheless, at the beginning of the course, they said to me that RL is an unsupervised learning approach because the agent performs the action, receives the response from the environment, and finally takes the more likely action, that is, with the highest Q value.
  
  But if I'm using deep learning in RL, for me, RL looks like a supervised learning approach. I'm a little confused about these things, could someone give me clarifications about them?
  
  [![Example from Online course](https://i.sstatic.net/JuKV7.png)](https://i.sstatic.net/JuKV7.png)
  
  **Tags:** deep-learning, reinforcement-learning, terminology, unsupervised-learning, supervised-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-05-04. Score: 3 | Accepted answer
  
  ### Supervised learning
  
  The supervised learning (SL) problem is formulated as follows.
  
  You are given a dataset $\mathcal{D} = \{(x\_i, y\_i)\_{i=1}^N$, which is assumed to be drawn i.i.d. from an **unknown** joint probability distribution $p(x, y)$, where $x\_i$ represents the $i$th input and $y\_i$ is the corresponding label. You choose a loss function $\mathcal{L}: V \times U \rightarrow \mathbb{R}$. Then your goal is to minimize the so-called *empirical risk*
  
  $$R\_{\mathcal{D}}[f]=\frac{1}{N} \sum\_{i=1}^N \mathcal{L}(x\_i, f(x\_i)) \tag{0}\label{0}$$
  
  with respect to $f$. In other words, you want to find the $f$ that minimizes the average above, which can also be formally written as
  $$
  f^\* = \operatorname{argmin}\_f R[f] \tag{1}\label{1}
  $$
  The problem \ref{1} is called the **empirical risk minimization** because it is a proxy problem for the **expected risk minimization** (but you can ignore this for now).
  
  ### Reinforcement learning
  
  In reinforcement learning, you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. At each time step $t$, the agent takes $a\_t$ in the state $s\_t$, receives a reward $r\_t$ from the environment and the agent and the environment move to another state $s\_{t+1}$.
  
  The goal of the agent is to maximize the **expected return**
  
  $$\mathbb{E}\left[ G\_t \right] = \mathbb{E}\left[ \sum\_{i=t+1}^\infty R\_i \right]$$
  
  where $t$ is the current time step (so we don't care about the past), $R\_i$ is a random variable that represents the probable reward at time step $i$, and $G\_t = \sum\_{i=t+1}^\infty R\_i $ is the so-called *return* (i.e. a sum of future rewards, in this case, starting from time step $t$), which is also a random variable.
  
  In this context, the most important job of the programmer is to define a function $\mathcal{R}(s, a)$, the reward function, which provides the *reinforcement* (or reward) signal to the RL agent. $\mathcal{R}(s, a)$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $a$ in the state $s$. (Note that $\mathcal{R}$ is different from $R\_i$, which is a random variable that represents the reward at time step $i$).
  
  ### What is the difference between SL and RL?
  
  In RL, you (the programmer) need to define the reward function $\mathcal{R}$ and you want to maximize the expected return. On the other hand, in SL you are given (or you collect) a dataset $\mathcal{D}$, you choose $\mathcal{L}$ in \ref{0}, and the goal is to find the function $f^\*$ that minimizes the empirical risk. So, these have different settings and goals, so they are different!
  
  However, every SL problem can be cast as an RL problem. See [this answer](https://ai.stackexchange.com/a/14168/2444). Similarly, in certain cases, you can formulate an RL as an SL problem. So, although the approaches are different, they are related.
  
  ### Is RL an unsupervised learning approach?
  
  In RL, you do **not** tell the agent what **action** it needs to take. You only say that the action that was taken was "bad", "good" or "so so". The agent needs to figure out which actions to take based on your feedback. In SL, you explicitly say that, for this input $x\_i$, the output should be $y\_i$.
  
  Some people may consider RL is an unsupervised learning approach, but I think this is wrong, because, in RL, the programmer still needs to define the reward function, so RL isn't totally unsupervised and it's also not totally supervised. For this reason, many people consider RL an approach that sits between UL and SL.
  
  ### What is deep learning?
  
  The term/expression deep learning (DL) refers to the use of deep neural networks (i.e. neural networks with many layers, where "many" can refer to more than 1 or 1000, i.e. it depends on the context) in machine learning, either supervised, unsupervised, or reinforcement learning. So, you can apply deep learning to SL, RL and UL. So, DL is not only restricted to SL.
  
  > **Comments:**
  > **username_3**: Nice answer. Since the OP asked about Deep Q-Learning, it might be worth clarifying the role of DL in that algorithm, along the following lines: a popular method for solving RL problems is Q-Learning, where one learns the Q-value function. In some RL problems with large state or action spaces, this function is difficult to compute directly, so we approximate it with a DNN—this is Deep Q-Learning. So there is no essential connection between RL and DL—it's just that some value functions are too complex to learn any other way (as far as we know).  
  > **username_2**: @username_3 Your explanation of DQN is correct, but I'd not go that far to say that there's no connection between RL and DL. They are both try to come up with solutions to the general learning problem and they can be combined, so there's a connection in this sense.  
  > **username_3**: Yes, you're right—it would be misleading to say that there is no connection between DL and RL (though I did say "no *essential* connection," by which I meant that it is possible to do RL without DL, contrary to what OP's instructor seemed to claim).  
  
  ---
  
  ### Answer by username_1 on 2020-05-04. Score: 2
  
  Reinforcement learning can be confusing in relation to supervised and unsupervised learning, especially for deep learning models. Here is the simple explanation make this clearer:
  
  ### Supervised Learning
  
  In the case of supervised learning, we are given a function that maps inputs to their desired outputs based on example input-output pairs. The basic setup is like this:
  
  * **Dataset:** There is a dataset with inputs $(x\_i)$ and corresponding labels $(y\_i)$.
  * **Objective:** The objective is to learn a function $(f(x))$ which makes the most accurate possible predictions about the labels $(y\_i)$.
  * **Loss Function:** You minimize a loss function $(L(f(x), y))$ over the dataset to find the best $(f)$.
  
  ### Reinforcement Learning
  
  In reinforcement learning, rather than being shown an example of what to do and being told this is the correct action for a given input, the agent has to learn from a scalar reward signal.
  
  * **Actions and States:** At every time step $(t)$ the agent is in state $(s\_t)$, takes action $(a\_t)$, and moves to a new state $(s\_{t+1})$.
  * **Rewards:** The agent receives a reward $(r\_t)$ after taking an action.
  * **Goal:** The goal is to maximize the cumulative reward, called the return $(G\_t = \sum\_{i=t+1}^\infty R\_i)$.
  
  ### The Role of Deep Learning in RL
  
  Deep learning is commonly used in RL to approximate functions such as the policy (what action to take for a given state) and the value function (expected return from a state). But the presence of deep learning does not fundamentally change RL.
  
  ### RL vs. Supervised Learning
  
  * **Feedback:** In supervised learning, you give explicit labels to each input. You provide rewards in RL: feedback signals that show the agent how to learn the best actions over time.
  * **Learning Process:** RL learns through exploration and exploitation, which is way more dynamic than static datasets used in supervised learning.
  
  ### Is RL Unsupervised Learning?
  
  Probably, some will argue that RL is unsupervised since it does not depend on labeled data like supervised learning. However, RL is not purely unsupervised because of the following two factors:
  
  * **Guidance:** The reward function is a form of guidance, indicating in some sense what the right thing to do is.
  * **Goal-oriented:** The agent learns to achieve an explicit goal—maximizing reward—a kind of supervision.
  
  Reinforcement learning is squarely situated within this learning paradigm:
  
  * It is not strictly supervised in that it does not use labeled input-output pairs in a straightforward way.
  * It is not purely unsupervised because it relies on rewards, a form of feedback that orients learning.
  
  Therefore, RL is often considered a separate machine learning category, which can leverage deep learning techniques to accomplish its goals.
  
  ### Additional Resources
  
  To gain deeper insights, consider watching the [Reinforcement Learning Course by David Silver](https://www.youtube.com/watch?v=2pWv7GOvuf0&t=2s), which takes a comprehensive approach to understanding the principles and implementations of RL.
  
  
  ---
  
  