Answers.Body: <p>Self-supervised learning is when you use some parts of the samples as labels for a task that requires a good degree of comprehension to be solved. I'll emphasize these two key points, before giving an example:</p>
  
  <ul>
  <li><p><em>Labels are extracted from the sample</em>, so they can be generated automatically, with some very simple algorithm (maybe just random selection).</p></li>
  <li><p><em>The task requires understanding</em>. This means that, in order to predict the output, the model has to extract some good patterns from the data, generating on the process a good representation.</p></li>
  </ul>
  
  <p>A very common case for semi-supervised learning takes place in natural language processing, when you need to solve a task but have few labeled data. In such cases, you need to learn a good representation or language model, so you take sentences and give your network self-supervision tasks like these:</p>
  
  <ul>
  <li><p>Ask the network to predict the next word in a sentence (which you know because you took it away).</p></li>
  <li><p>Mask a word and ask the network to predict which word goes there (which you know because you had to mask it).</p></li>
  <li><p>Change the word for a random one (that probably doesn't make sense) and ask the network which word is wrong.</p></li>
  </ul>
  
  <p>As you can see, these tasks are fairly simple to formulate and the labels are part of the same sample, but they require a certain understanding of the context to be solved.</p>
  
  <p>And it's always like this: alter your data in some way, generating the label in the process, and ask the model something related to that transformation. If the task requires enough understanding of the data, you'll have success.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-02T22:06:05.900Z
Answers.LastActivityDate: 2019-08-02T22:06:05.900Z
Answers.UserDisplayName: David
Answers.Body: <p>Self-supervised visual recognition is often applied to representation learning. Here we first learn features on unlabeled data (representation learning), and then learn the real model on features extracted from the labeled data. This especially makes sense when we have a lot of unlabeled data and few labeled data.</p>
  
  <p>The features can be learned by solving so called <em>pretext</em> tasks. Examples of pretext tasks are to predict rotation of a jittered image, to recognize jittered instances of a same image, or to predict spatial relationship of image patches.</p>
  
  <p>A nice overview and interesting results can be found in <a href="https://arxiv.org/pdf/1901.09005.pdf" rel="noreferrer">this</a> recent paper.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-25T12:08:31.247Z
Answers.Text: Do you call the entire setup "self-supervised learning" or only the representation learning part? What is the difference between the terms "self-supervised learning" and "representation learning" then?
Answers.UserDisplayName: Make42
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-26T13:05:05.633Z
Answers.Text: Self-supervised learning is a way to achieve representation learning. Some other ways to achieve the same goal are supervised learning and unsupervised learning.
Answers.UserDisplayName: ssegvic
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-20T14:30:03.233Z
Answers.LastActivityDate: 2019-02-20T14:30:03.233Z
Answers.UserDisplayName: ssegvic
Answers.Body: <h2>Introduction</h2>
  <p>The term <em>self-supervised learning</em> (SSL) has been used (sometimes differently) in different contexts and fields, such as representation learning [<a href="https://arxiv.org/pdf/1708.07860.pdf" rel="noreferrer">1</a>], neural networks, robotics [<a href="https://arxiv.org/pdf/1809.07207.pdf" rel="noreferrer">2</a>], natural language processing, and reinforcement learning. In all cases, the basic idea is to <strong>automatically</strong> generate some kind of supervisory signal to solve some task (typically, to learn representations of the data or to automatically label a dataset).</p>
  <p>I will describe what SSL means more specifically in three contexts: representation learning, neural networks and robotics.</p>
  <h2>Representation learning</h2>
  <p>The term self-supervised learning has been widely used to refer to techniques that do not use human-annotated datasets to learn (visual) representations of the data (i.e. representation learning).</p>
  <h3>Example</h3>
  <p>In [<a href="https://arxiv.org/pdf/1708.07860.pdf" rel="noreferrer">1</a>], two patches are randomly selected and cropped from an unlabelled image and the goal is to predict the relative position of the two patches. Of course, we have the relative position of the two patches once you have chosen them (i.e. we can keep track of their centers), so, in this case, this is the automatically generated supervisory signal. The idea is that, to solve this task (known as a <strong>pretext</strong> or <strong>auxiliary</strong> task in the literature [<a href="https://arxiv.org/pdf/1905.01235.pdf" rel="noreferrer">3</a>, <a href="https://arxiv.org/pdf/1505.05192.pdf" rel="noreferrer">4</a>, <a href="https://arxiv.org/pdf/1803.07728.pdf" rel="noreferrer">5</a>, <a href="https://arxiv.org/pdf/1901.09005.pdf" rel="noreferrer">6</a>]), the neural network needs to learn features in the images. These learned representations can then be used to solve the so-called <strong>downstream</strong> tasks, i.e. the tasks you are interested in (e.g. object detection or semantic segmentation).</p>
  <p>So, you first learn representations of the data (by SSL pre-training), then you can transfer these learned representations to solve a task that you actually want to solve, and you can do this by fine-tuning the neural network that contains the learned representations on a labeled (but smaller dataset), i.e. you can use SSL for transfer learning.</p>
  <p>This example is similar to the example given <a href="https://ai.stackexchange.com/a/13749/2444">in this other answer</a>.</p>
  <h3>Neural networks</h3>
  <p>Some neural networks, for example, <strong>autoencoders</strong> (AE) [<a href="https://link.springer.com/chapter/10.1007/978-3-319-46672-9_10" rel="noreferrer">7</a>] are sometimes called self-supervised learning tools. In fact, you can train AEs without images that have been manually labeled by a human. More concretely, consider a de-noising AE, whose goal is to reconstruct the original image when given a noisy version of it. During training, you actually have the original image, given that you have a dataset of uncorrupted images and you just corrupt these images with some noise, so you can calculate some kind of distance between the original image and the noisy one, where the original image is the supervisory signal. In this sense, AEs are self-supervised learning tools, but it's more common to say that AEs are unsupervised learning tools, so SSL has also been used to refer to unsupervised learning techniques.</p>
  <h2>Robotics</h2>
  <p>In [<a href="https://arxiv.org/pdf/1809.07207.pdf" rel="noreferrer">2</a>], the training data is <strong>automatically</strong> but approximately labeled by finding and exploiting the relations or correlations between inputs coming from different sensor modalities (and this technique is called SSL by the authors). So, as opposed to representation learning or auto-encoders, in this case, an actual labeled dataset is produced automatically.</p>
  <h3>Example</h3>
  <p>Consider a robot that is equipped with a <em>proximity sensor</em> (which is a <em>short-range</em> sensor capable of detecting objects in front of the robot at short distances) and a <em>camera</em> (which is <em>long-range</em> sensor, but which does not provide a direct way of detecting objects). You can also assume that this robot is capable of performing <a href="https://en.wikipedia.org/wiki/Odometry" rel="noreferrer">odometry</a>. An example of such a robot is <a href="https://github.com/jeguzzi/mighty-thymio" rel="noreferrer">Mighty Thymio</a>.</p>
  <p>Consider now the task of detecting objects in front of the robot at longer ranges than the range the proximity sensor allows. In general, we could train a CNN to achieve that. However, to train such CNN, in supervised learning, we would first need a labelled dataset, which contains labelled images (or videos), where the labels could e.g. be &quot;object in the image&quot; or &quot;no object in the image&quot;. In supervised learning, this dataset would need to be manually labelled by a human, which clearly would require a lot of work.</p>
  <p>To overcome this issue, we can use a self-supervised learning approach. In this example, the basic idea is to associate the output of the proximity sensors at a time step <span class="math-container">$t' &gt; t$</span> with the output of the camera at time step <span class="math-container">$t$</span> (a smaller time step than <span class="math-container">$t'$</span>).</p>
  <p>More specifically, suppose that the robot is <em>initially</em> at coordinates <span class="math-container">$(x, y)$</span> (on the plane), at time step <span class="math-container">$t$</span>. At this point, we still do not have enough info to label the output of the camera (at the same time step <span class="math-container">$t$</span>). Suppose now that, at time <span class="math-container">$t'$</span>, the robot is at position <span class="math-container">$(x', y')$</span>. At time step <span class="math-container">$t'$</span>, the output of the proximity sensor will e.g. be &quot;object in front of the robot&quot; or &quot;no object in front of the robot&quot;. Without loss of generality, suppose that the output of the proximity sensor at <span class="math-container">$t' &gt; t$</span> is &quot;no object in front of the robot&quot;, then the label associated with the output of the camera (an image frame) at time <span class="math-container">$t$</span> will be &quot;no object in front of the robot&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-20T21:13:31.900Z
Answers.Text: See also the paper [Learning to Fly by Crashing](https://arxiv.org/abs/1704.05588). It turns out proximity sensor is optional :D
Answers.UserDisplayName: P.Windridge
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-22T15:54:31.267Z
Answers.Text: "Calculate some kind of distance between the original image and the noisy one" - don't you mean calculate distance between the original image and the *de-noised* one (after running the noisy one through the autoencoder)?
Answers.UserDisplayName: jskattt797
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-09T18:28:53.737Z
Answers.Text: @nbro Is it correct so say that self-supervised learning is a learning paradigm where the labels require no human annotation?
Answers.UserDisplayName: Antonios Sarikas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-10T01:37:22.260Z
Answers.Text: @adosar Yes, I'd say so. So, the humans don't have to directly label the inputs, but they still devise the ideas of how the algorithm automatically extracts the labels (e.g. in the robotic case that I mention in my answer) or how the algorithm learns (e.g. the human may decide to rotate the images at certain random angle and keep track of this info, then use this as a pre-text task).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-16T20:02:58.273Z
Answers.LastActivityDate: 2020-08-01T13:58:51.847Z
Answers.LastEditDate: 2020-08-01T13:58:51.847Z
Answers.UserDisplayName: nbro
Body: <p>What is self-supervised learning in machine learning? How is it different from supervised learning?</p>
  
Tags: <machine-learning><comparison><supervised-learning><self-supervised-learning><representation-learning>
Title: What is self-supervised learning in machine learning?
UserDisplayName: nbro
ThreadText: # What is self-supervised learning in machine learning?
  
  (Asked by: username_0 on 2019-02-16)
  
  What is self-supervised learning in machine learning? How is it different from supervised learning?
  
  **Tags:** machine-learning, comparison, supervised-learning, self-supervised-learning, representation-learning
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_0 on 2019-02-16. Score: 97 | Accepted answer
  
  ## Introduction
  
  The term *self-supervised learning* (SSL) has been used (sometimes differently) in different contexts and fields, such as representation learning [[1](https://arxiv.org/pdf/1708.07860.pdf)], neural networks, robotics [[2](https://arxiv.org/pdf/1809.07207.pdf)], natural language processing, and reinforcement learning. In all cases, the basic idea is to **automatically** generate some kind of supervisory signal to solve some task (typically, to learn representations of the data or to automatically label a dataset).
  
  I will describe what SSL means more specifically in three contexts: representation learning, neural networks and robotics.
  
  ## Representation learning
  
  The term self-supervised learning has been widely used to refer to techniques that do not use human-annotated datasets to learn (visual) representations of the data (i.e. representation learning).
  
  ### Example
  
  In [[1](https://arxiv.org/pdf/1708.07860.pdf)], two patches are randomly selected and cropped from an unlabelled image and the goal is to predict the relative position of the two patches. Of course, we have the relative position of the two patches once you have chosen them (i.e. we can keep track of their centers), so, in this case, this is the automatically generated supervisory signal. The idea is that, to solve this task (known as a **pretext** or **auxiliary** task in the literature [[3](https://arxiv.org/pdf/1905.01235.pdf), [4](https://arxiv.org/pdf/1505.05192.pdf), [5](https://arxiv.org/pdf/1803.07728.pdf), [6](https://arxiv.org/pdf/1901.09005.pdf)]), the neural network needs to learn features in the images. These learned representations can then be used to solve the so-called **downstream** tasks, i.e. the tasks you are interested in (e.g. object detection or semantic segmentation).
  
  So, you first learn representations of the data (by SSL pre-training), then you can transfer these learned representations to solve a task that you actually want to solve, and you can do this by fine-tuning the neural network that contains the learned representations on a labeled (but smaller dataset), i.e. you can use SSL for transfer learning.
  
  This example is similar to the example given [in this other answer](https://ai.stackexchange.com/a/13749/2444).
  
  ### Neural networks
  
  Some neural networks, for example, **autoencoders** (AE) [[7](https://link.springer.com/chapter/10.1007/978-3-319-46672-9_10)] are sometimes called self-supervised learning tools. In fact, you can train AEs without images that have been manually labeled by a human. More concretely, consider a de-noising AE, whose goal is to reconstruct the original image when given a noisy version of it. During training, you actually have the original image, given that you have a dataset of uncorrupted images and you just corrupt these images with some noise, so you can calculate some kind of distance between the original image and the noisy one, where the original image is the supervisory signal. In this sense, AEs are self-supervised learning tools, but it's more common to say that AEs are unsupervised learning tools, so SSL has also been used to refer to unsupervised learning techniques.
  
  ## Robotics
  
  In [[2](https://arxiv.org/pdf/1809.07207.pdf)], the training data is **automatically** but approximately labeled by finding and exploiting the relations or correlations between inputs coming from different sensor modalities (and this technique is called SSL by the authors). So, as opposed to representation learning or auto-encoders, in this case, an actual labeled dataset is produced automatically.
  
  ### Example
  
  Consider a robot that is equipped with a *proximity sensor* (which is a *short-range* sensor capable of detecting objects in front of the robot at short distances) and a *camera* (which is *long-range* sensor, but which does not provide a direct way of detecting objects). You can also assume that this robot is capable of performing [odometry](https://en.wikipedia.org/wiki/Odometry). An example of such a robot is [Mighty Thymio](https://github.com/jeguzzi/mighty-thymio).
  
  Consider now the task of detecting objects in front of the robot at longer ranges than the range the proximity sensor allows. In general, we could train a CNN to achieve that. However, to train such CNN, in supervised learning, we would first need a labelled dataset, which contains labelled images (or videos), where the labels could e.g. be "object in the image" or "no object in the image". In supervised learning, this dataset would need to be manually labelled by a human, which clearly would require a lot of work.
  
  To overcome this issue, we can use a self-supervised learning approach. In this example, the basic idea is to associate the output of the proximity sensors at a time step $t' > t$ with the output of the camera at time step $t$ (a smaller time step than $t'$).
  
  More specifically, suppose that the robot is *initially* at coordinates $(x, y)$ (on the plane), at time step $t$. At this point, we still do not have enough info to label the output of the camera (at the same time step $t$). Suppose now that, at time $t'$, the robot is at position $(x', y')$. At time step $t'$, the output of the proximity sensor will e.g. be "object in front of the robot" or "no object in front of the robot". Without loss of generality, suppose that the output of the proximity sensor at $t' > t$ is "no object in front of the robot", then the label associated with the output of the camera (an image frame) at time $t$ will be "no object in front of the robot".
  
  > **Comments:**
  > **username_4**: See also the paper [Learning to Fly by Crashing](https://arxiv.org/abs/1704.05588). It turns out proximity sensor is optional :D  
  > **username_5**: "Calculate some kind of distance between the original image and the noisy one" - don't you mean calculate distance between the original image and the *de-noised* one (after running the noisy one through the autoencoder)?  
  > **username_6**: @username_0 Is it correct so say that self-supervised learning is a learning paradigm where the labels require no human annotation?  
  > **username_0**: @adosar Yes, I'd say so. So, the humans don't have to directly label the inputs, but they still devise the ideas of how the algorithm automatically extracts the labels (e.g. in the robotic case that I mention in my answer) or how the algorithm learns (e.g. the human may decide to rotate the images at certain random angle and keep track of this info, then use this as a pre-text task).  
  
  ---
  
  ### Answer by username_1 on 2019-08-02. Score: 19
  
  Self-supervised learning is when you use some parts of the samples as labels for a task that requires a good degree of comprehension to be solved. I'll emphasize these two key points, before giving an example:
  
  * *Labels are extracted from the sample*, so they can be generated automatically, with some very simple algorithm (maybe just random selection).
  * *The task requires understanding*. This means that, in order to predict the output, the model has to extract some good patterns from the data, generating on the process a good representation.
  
  A very common case for semi-supervised learning takes place in natural language processing, when you need to solve a task but have few labeled data. In such cases, you need to learn a good representation or language model, so you take sentences and give your network self-supervision tasks like these:
  
  * Ask the network to predict the next word in a sentence (which you know because you took it away).
  * Mask a word and ask the network to predict which word goes there (which you know because you had to mask it).
  * Change the word for a random one (that probably doesn't make sense) and ask the network which word is wrong.
  
  As you can see, these tasks are fairly simple to formulate and the labels are part of the same sample, but they require a certain understanding of the context to be solved.
  
  And it's always like this: alter your data in some way, generating the label in the process, and ask the model something related to that transformation. If the task requires enough understanding of the data, you'll have success.
  
  
  ---
  
  ### Answer by username_2 on 2019-02-20. Score: 7
  
  Self-supervised visual recognition is often applied to representation learning. Here we first learn features on unlabeled data (representation learning), and then learn the real model on features extracted from the labeled data. This especially makes sense when we have a lot of unlabeled data and few labeled data.
  
  The features can be learned by solving so called *pretext* tasks. Examples of pretext tasks are to predict rotation of a jittered image, to recognize jittered instances of a same image, or to predict spatial relationship of image patches.
  
  A nice overview and interesting results can be found in [this](https://arxiv.org/pdf/1901.09005.pdf) recent paper.
  
  > **Comments:**
  > **username_3**: Do you call the entire setup "self-supervised learning" or only the representation learning part? What is the difference between the terms "self-supervised learning" and "representation learning" then?  
  > **username_2**: Self-supervised learning is a way to achieve representation learning. Some other ways to achieve the same goal are supervised learning and unsupervised learning.  
  
  ---
  
  