Answers.Body: <p>In <em>self-supervised learning</em> (SSL) you use your own inputs <span class="math-container">$x$</span> (or a modification, e.g. a crop or with data augmentation applied) as the supervision. Instead, in <em>unsupervised learning</em> (UL) there is no supervision at all.</p>
  <p>To clarify, both SSL and UL have in common the fact that <strong>the targets are missing</strong>. UL has no explicit supervision, while SSL replaces the targets with the inputs <span class="math-container">$x$</span>, recovering 'supervision'.</p>
  <ul>
  <li>SSL is mostly used for <em>pre-training</em>, and <em>representation learning</em>. So to bootstrap some model on a later downstream task.</li>
  <li>UL, at least in classical ML, for <em>density estimation</em>, <em>dimensionality reduction</em> and <em>clustering</em>.</li>
  </ul>
  <blockquote>
  <p>An important thing is not to confuse self-supervised with <strong>semi-supervised</strong> or <strong>weakly-supervised</strong>: the latter two (semi- and weak-) refer to the fact that in a dataset <span class="math-container">$D$</span> some examples <span class="math-container">$x$</span> are not labeled, but the <span class="math-container">$y$</span> exist.</p>
  </blockquote>
  <p>So, you can see SSL at the intersection between supervised and unsupervised learning. Actually, things got even more shaded in modern unsupervised deep learning methods that tend to mix approaches from both SSL and UL, like an AE that also have a density estimation head for example. Or even embeddings that are first learned by SSL and then fine-tuned for clustering in an unsupervised manner.</p>
  <p>An unusual example is maybe <em>unsupervised reinforcement learning</em>, in which you maximize usually an entropy objective (e.g. on visited states) as a pre-training step to favor exploration.</p>
  <blockquote>
  <p>What is an example of unsupervised learning that is definitely not self-supervised learning?</p>
  </blockquote>
  <p>Density estimation, dimensionality reduction (e.g. PCA, t-SNE), and clustering (K-means), at least seen from a classical ML prospective are completely unsupervised: e.g. PCA tries just to preserve variance.  Indeed, in DL things tend to blurry: e.g. you can use a V/AE for dimensionality reduction too.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-27T11:01:43.223Z
Answers.Text: Well if we see PCA as trying to minimize the reconstruction error, couldn't we say that PCA is SSL?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-28T09:46:03.217Z
Answers.Text: @adosar I think modern DL is challenging common ML notions and assumptions. So, if you see PCA as representation learning (i.e., to learn a lower-dim representation that still holds all the relevant information), which can be implemented by SSL, then the answer is probably yes. I think that with DL, the differences between learning paradigms are getting blurred.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-08T09:12:57.500Z
Answers.LastActivityDate: 2024-07-28T09:41:24.243Z
Answers.LastEditDate: 2024-07-28T09:41:24.243Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <h3>TL;DR</h3>
  <p>While both methods learn from data without human-annotated labels, the primary difference lies in the way they use the data:</p>
  <ul>
  <li><strong>Self-supervised</strong> learning makes use of the structure within the data to generate its own labels.</li>
  <li><strong>Unsupervised learning</strong> seeks to uncover hidden patterns or structures within the data itself.</li>
  </ul>
  <hr />
  <h3>Self-supervised</h3>
  <pre><code>Input: &quot;The quick brown fox jumps over the _____&quot;
  Prediction: &quot;lazy&quot;
  Actual: &quot;lazy&quot;
  </code></pre>
  <p>In this case, the model is learning to predict the next word (&quot;lazy&quot;) based on the input (&quot;The quick brown fox jumps over the&quot;). It's a self-supervised task because the label for training (the word &quot;lazy&quot;) is part of the data itself.</p>
  <h3>Unsupervised</h3>
  <p>For unsupervised learning, let's consider an example with text clustering. Suppose we have multiple sentences, including multiple instances of our sentence, and we want to group similar sentences together:</p>
  <pre><code>1. &quot;The quick brown fox jumps over the lazy&quot;
  2. &quot;A fast brown fox jumps over the lazy&quot;
  3. &quot;The quick brown dog runs under the active&quot;
  4. &quot;An energetic brown dog runs under the quick&quot;
  5. &quot;The quick brown fox jumps over the lazy&quot;
  6. &quot;A swift brown fox leaps over the lazy&quot;
  
  Clusters after unsupervised learning:
      Cluster 1: 1, 2, 5, 6 
      Cluster 2: 3, 4
  </code></pre>
  <p>In this case, the model is learning to group similar sentences together without any explicit labels. Note that this is a highly simplified example, and real-world text clustering tasks would involve much more complex datasets and models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-02T17:41:31.923Z
Answers.LastActivityDate: 2023-06-02T17:50:28.890Z
Answers.LastEditDate: 2023-06-02T17:50:28.890Z
Answers.UserDisplayName: ofou
Answers.Body: <p>Self-supervised learning is one approach to unsupervised learning.  There are other approaches to unsupervised learning, too.  In both cases, we have a dataset of instances with no labels, and we're trying to use them to learn a classifier.</p>
  <p>Unsupervised learning includes any method for learning from unlabelled samples.  Self-supervised learning is one specific class of methods to learn from unlabelled samples.  Typically, self-supervised learning identifies some secondary task where labels can be automatically obtained, and then trains the network to do well on the secondary task.</p>
  <p>One example of a secondary task is to predict whether an image has been flipped to be upside-down or not.  Effectively, the assumption is that all (or most) natural photographs are taken by a photographer from an upright position, so we can automatically construct labels for all images in our dataset, as they're likely all right-side-up.  (Moreover, we can create upside-down versions of all images in the dataset and automatically create labels of them: they're likely all upside-down.)</p>
  <p>The concept behind self-supervised learning is that, hopefully, the classifier will have to learn something non-trivial about image semantics to be able to solve the secondary task.  Thus, such a classifier might be a good starting point for the primary task, too.</p>
  <p>Often, self-supervised learning is combined with supervised learning.  For instance, we might have a small set of labelled images (labelled for the primary task we ultimately care about) and a large set of unlabelled images, and the classifier is trained to minimize a hybrid loss, which is the sum of a supervised loss on the labelled images and a self-supervised loss on the unlabelled images.  This can be considered an instance of semi-supervised learning.  Or, another example setting is that sometimes people learn a classifier on a large unlabelled dataset using self-supervised learning, then they fine-tune that classifier on a small labelled dataset for the primary task.  This can be considered a form of transfer learning.</p>
  <p>Empirically, this approach often leads to classifiers with better performance on the primary task than just using supervised learning on the labelled dataset.  Or, it leads to classifiers that are more robust (to data corruption, distribution shift, etc.) than simply using supervised learning on the labelled dataset.</p>
  <p>There are many forms of self-supervised learning, and techniques depend on the specific problem domain.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-09T05:38:58.387Z
Answers.LastActivityDate: 2023-05-09T05:38:58.387Z
Answers.UserDisplayName: D.W.
Answers.Body: <p>It's misleading to think of these as separate learning paradigms. Self-supervised learning IS an unsupervised learning algorithm that uses certain methods to derive learning signals without explicit labels.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-01T05:25:26.610Z
Answers.LastActivityDate: 2024-02-01T05:25:26.610Z
Answers.UserDisplayName: Jing
Body: <p>What is the difference between <a href="https://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning?noredirect=1&amp;lq=1">self-supervised</a> and unsupervised learning? The terms logically overlap (and maybe self-supervised learning is a subset of unsupervised learning?), but I cannot pinpoint exactly what that difference is.</p>
  <p>What are the commonly agreed-upon 'definitions' of these terms? What is an example of unsupervised learning that is definitely not self-supervised learning?</p>
  
Tags: <definitions><unsupervised-learning><self-supervised-learning>
Title: What is the difference between self-supervised and unsupervised learning?
UserDisplayName: Robin van Hoorn
ThreadText: # What is the difference between self-supervised and unsupervised learning?
  
  (Asked by: username_0 on 2023-05-07)
  
  What is the difference between [self-supervised](https://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning?noredirect=1&lq=1) and unsupervised learning? The terms logically overlap (and maybe self-supervised learning is a subset of unsupervised learning?), but I cannot pinpoint exactly what that difference is.
  
  What are the commonly agreed-upon 'definitions' of these terms? What is an example of unsupervised learning that is definitely not self-supervised learning?
  
  **Tags:** definitions, unsupervised-learning, self-supervised-learning
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2023-05-08. Score: 13
  
  In *self-supervised learning* (SSL) you use your own inputs $x$ (or a modification, e.g. a crop or with data augmentation applied) as the supervision. Instead, in *unsupervised learning* (UL) there is no supervision at all.
  
  To clarify, both SSL and UL have in common the fact that **the targets are missing**. UL has no explicit supervision, while SSL replaces the targets with the inputs $x$, recovering 'supervision'.
  
  * SSL is mostly used for *pre-training*, and *representation learning*. So to bootstrap some model on a later downstream task.
  * UL, at least in classical ML, for *density estimation*, *dimensionality reduction* and *clustering*.
  
  > An important thing is not to confuse self-supervised with **semi-supervised** or **weakly-supervised**: the latter two (semi- and weak-) refer to the fact that in a dataset $D$ some examples $x$ are not labeled, but the $y$ exist.
  
  So, you can see SSL at the intersection between supervised and unsupervised learning. Actually, things got even more shaded in modern unsupervised deep learning methods that tend to mix approaches from both SSL and UL, like an AE that also have a density estimation head for example. Or even embeddings that are first learned by SSL and then fine-tuned for clustering in an unsupervised manner.
  
  An unusual example is maybe *unsupervised reinforcement learning*, in which you maximize usually an entropy objective (e.g. on visited states) as a pre-training step to favor exploration.
  
  > What is an example of unsupervised learning that is definitely not self-supervised learning?
  
  Density estimation, dimensionality reduction (e.g. PCA, t-SNE), and clustering (K-means), at least seen from a classical ML prospective are completely unsupervised: e.g. PCA tries just to preserve variance. Indeed, in DL things tend to blurry: e.g. you can use a V/AE for dimensionality reduction too.
  
  > **Comments:**
  > **username_2**: Well if we see PCA as trying to minimize the reconstruction error, couldn't we say that PCA is SSL?  
  > **username_1**: @adosar I think modern DL is challenging common ML notions and assumptions. So, if you see PCA as representation learning (i.e., to learn a lower-dim representation that still holds all the relevant information), which can be implemented by SSL, then the answer is probably yes. I think that with DL, the differences between learning paradigms are getting blurred.  
  
  ---
  
  ### Answer by username_3 on 2023-06-02. Score: 7
  
  ### TL;DR
  
  While both methods learn from data without human-annotated labels, the primary difference lies in the way they use the data:
  
  * **Self-supervised** learning makes use of the structure within the data to generate its own labels.
  * **Unsupervised learning** seeks to uncover hidden patterns or structures within the data itself.
  
  ---
  
  ### Self-supervised
  
  ```
  Input: "The quick brown fox jumps over the _____"
  Prediction: "lazy"
  Actual: "lazy"
  
  ```
  
  In this case, the model is learning to predict the next word ("lazy") based on the input ("The quick brown fox jumps over the"). It's a self-supervised task because the label for training (the word "lazy") is part of the data itself.
  
  ### Unsupervised
  
  For unsupervised learning, let's consider an example with text clustering. Suppose we have multiple sentences, including multiple instances of our sentence, and we want to group similar sentences together:
  
  ```
  1. "The quick brown fox jumps over the lazy"
  2. "A fast brown fox jumps over the lazy"
  3. "The quick brown dog runs under the active"
  4. "An energetic brown dog runs under the quick"
  5. "The quick brown fox jumps over the lazy"
  6. "A swift brown fox leaps over the lazy"
  
  Clusters after unsupervised learning:
      Cluster 1: 1, 2, 5, 6 
      Cluster 2: 3, 4
  
  ```
  
  In this case, the model is learning to group similar sentences together without any explicit labels. Note that this is a highly simplified example, and real-world text clustering tasks would involve much more complex datasets and models.
  
  
  ---
  
  ### Answer by username_4 on 2023-05-09. Score: 2
  
  Self-supervised learning is one approach to unsupervised learning. There are other approaches to unsupervised learning, too. In both cases, we have a dataset of instances with no labels, and we're trying to use them to learn a classifier.
  
  Unsupervised learning includes any method for learning from unlabelled samples. Self-supervised learning is one specific class of methods to learn from unlabelled samples. Typically, self-supervised learning identifies some secondary task where labels can be automatically obtained, and then trains the network to do well on the secondary task.
  
  One example of a secondary task is to predict whether an image has been flipped to be upside-down or not. Effectively, the assumption is that all (or most) natural photographs are taken by a photographer from an upright position, so we can automatically construct labels for all images in our dataset, as they're likely all right-side-up. (Moreover, we can create upside-down versions of all images in the dataset and automatically create labels of them: they're likely all upside-down.)
  
  The concept behind self-supervised learning is that, hopefully, the classifier will have to learn something non-trivial about image semantics to be able to solve the secondary task. Thus, such a classifier might be a good starting point for the primary task, too.
  
  Often, self-supervised learning is combined with supervised learning. For instance, we might have a small set of labelled images (labelled for the primary task we ultimately care about) and a large set of unlabelled images, and the classifier is trained to minimize a hybrid loss, which is the sum of a supervised loss on the labelled images and a self-supervised loss on the unlabelled images. This can be considered an instance of semi-supervised learning. Or, another example setting is that sometimes people learn a classifier on a large unlabelled dataset using self-supervised learning, then they fine-tune that classifier on a small labelled dataset for the primary task. This can be considered a form of transfer learning.
  
  Empirically, this approach often leads to classifiers with better performance on the primary task than just using supervised learning on the labelled dataset. Or, it leads to classifiers that are more robust (to data corruption, distribution shift, etc.) than simply using supervised learning on the labelled dataset.
  
  There are many forms of self-supervised learning, and techniques depend on the specific problem domain.
  
  
  ---
  
  ### Answer by username_5 on 2024-02-01. Score: 0
  
  It's misleading to think of these as separate learning paradigms. Self-supervised learning IS an unsupervised learning algorithm that uses certain methods to derive learning signals without explicit labels.
  
  
  ---
  
  