Answers.Body: <blockquote>
  <p><em><strong>How can I generate the target label from the other data in the dataset?</strong></em></p>
  </blockquote>
  <p>If you are asking how you can create the learning signal in SSL, when given an unlabelled dataset, for learning representations of these unlabelled data, then there is <strong>no general answer</strong>. The answer depends on the type of data that you have (which can be e.g. textual or visual), and which features do you think you want to learn or can be learned from your unlabelled data. <a href="https://arxiv.org/pdf/1902.06162.pdf" rel="noreferrer">This paper</a> and other answers to <a href="https://ai.stackexchange.com/q/10623/2444">this question</a> provide some examples of how that can be done (depending on the type of data). Below, I also provide an example.</p>
  <p>Let me try to explain this more in detail.</p>
  <p>Let's assume that you have <em>both</em></p>
  <ol>
  <li><p>an <strong>unlabelled</strong> dataset <span class="math-container">$U = \{ u_i \}_{i=1}^m$</span> and</p>
  </li>
  <li><p>a <strong>labelled</strong> dataset <span class="math-container">$D = \{(x_i, y_i) \}_{i=1}^n$</span></p>
  </li>
  </ol>
  <p>where we may have <span class="math-container">$m \gg n$</span> (although this is not a strict requirement), i.e. you may have a lot more unlabelled data than labelled data (this can easily be the case, given that, in general, manual data annotation is expensive/laborious). Let's say that your ultimate task is to perform object recognition (or classification). Let's call this task the <strong>downstream task</strong>. So, you may think that <span class="math-container">$x_i$</span> and <span class="math-container">$u_i$</span> are images and <span class="math-container">$y_i$</span> are labels, like &quot;cat&quot; or &quot;dog&quot; (let's say that you want to differentiate between cats and dogs).</p>
  <p>You want to solve this downstream task by supervised learning with <span class="math-container">$D$</span>. However, given that your labeled dataset is not big enough, you may think that training a neural network from scratch (i.e. by randomly initializing its weights) with <span class="math-container">$D$</span> may not lead to good performance. So, you think that it could be useful to start training from a pre-trained model that already contains useful representations of data similar to your labeled data, i.e. to perform <a href="https://www.tensorflow.org/tutorials/images/transfer_learning" rel="noreferrer"><strong>transfer learning</strong></a>. To pre-train such a model, you could use SSL.</p>
  <p>So, to solve your downstream task with SSL, there are <strong>2 different steps</strong></p>
  <ol>
  <li><p><strong>Self-supervised learning (SSL)</strong>: learn representations of your images <span class="math-container">$u_i \in U$</span> by training a neural network <span class="math-container">$M$</span> with <span class="math-container">$U$</span> to solve a so-called <strong>pretext</strong> (or <strong>auxiliary</strong> task); there are many pre-text tasks: you can find many examples <a href="https://arxiv.org/pdf/1902.06162.pdf" rel="noreferrer">here</a>, <a href="https://ai.stackexchange.com/a/20393/2444">here</a> and <a href="https://ai.stackexchange.com/q/10623/2444">here</a> (see example below too);</p>
  </li>
  <li><p><strong>Supervised learning (SL)</strong> by <strong>transfer learning</strong>: fine-tune <span class="math-container">$M$</span> with <span class="math-container">$D$</span> (the labeled dataset), in a supervised way; this task is known as <strong>downstream task</strong> (as stated above)</p>
  </li>
  </ol>
  <p>In this process, there are <strong>2 different labels</strong>.</p>
  <ul>
  <li><p>In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your <strong>unlabelled</strong> dataset <span class="math-container">$U$</span> contains high-resolution images (i.e. <span class="math-container">$u_i \in U$</span> are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images to create other images. Let <span class="math-container">$v_i$</span> be the low-resolution image created from the high-resolution image <span class="math-container">$u_i \in U$</span>, then the training pair to your neural network <span class="math-container">$M$</span> is <span class="math-container">$(v_i, u_i) \in U'$</span>, where <span class="math-container">$u_i$</span> is the label (which is the original high-resolution image) and <span class="math-container">$U'$</span> the labeled dataset automatically generated (i.e. with the algorithm I've just explained).</p>
  <p>So, these labels <span class="math-container">$u_i$</span> (high-resolution images) are semantically different than <span class="math-container">$y_i$</span> (&quot;cat&quot; or &quot;dog&quot;) in the pairs <span class="math-container">$(x_i, y_i) \in D$</span>. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.</p>
  </li>
  <li><p>In step 2, you use the labeled dataset <span class="math-container">$D$</span>, which has been typically annotated (or labeled) by a human. As stated above, this dataset contains pairs <span class="math-container">$(x_i, y_i)$</span>, where <span class="math-container">$y_i$</span> is, for example, the label &quot;cat&quot; or &quot;dog&quot;.</p>
  <p>In this step, the pre-trained model <span class="math-container">$M$</span>, with the SSL technique,  can be fine-tuned with <span class="math-container">$D$</span> in a supervised fashion. Given that we start with a pre-trained model <span class="math-container">$M$</span>, we are effectively performing transfer learning.</p>
  </li>
  </ul>
  <p>Note that SSL can also refer to something (slightly) different than what has been explained in this answer. See <a href="https://ai.stackexchange.com/a/10624/2444">my other answer</a> for more details. Moreover, note that you can perform representation learning with SSL without necessarily solving a downstream task later, which may also not be an SL task (in the example above, I've described a downstream task that is an SL task only for simplicity).</p>
  <p>If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques (such as <a href="https://github.com/google/revisiting-self-supervised" rel="noreferrer">this</a>) for more inspiration.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-04T11:38:19.113Z
Answers.LastActivityDate: 2020-11-20T12:47:33.780Z
Answers.LastEditDate: 2020-11-20T12:47:33.780Z
Answers.UserDisplayName: nbro
Body: <p>I've been reading a lot lately about self-supervised learning and I didn't understand very well how to generate the desired label for a given image.</p>
  <p>Let's say that I have an image classification task, and I have very little labeled data.</p>
  <p><em><strong>How can I generate the target label from the other data in the dataset?</strong></em></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T00:45:41.717Z
Comments.Text: Hi, I am also working on SSL task or more learning how to do it. Is it please possible, if you show me the way you did it please? :)
Comments.UserDisplayName: Timco Vanco
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T03:42:59.950Z
Comments.Text: @TimcoVanco See my [**updated** answer below](https://ai.stackexchange.com/a/22860/2444). It should clarify your doubts, at this point, I believe.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-20T12:50:38.553Z
Comments.Text: You should accept my answer below (by clicking on the checkmark), if it answers your question. It should answer your question, at this point. If not, please, let me know what is still unclear.
Comments.UserDisplayName: nbro
Tags: <deep-learning><datasets><supervised-learning><self-supervised-learning><representation-learning>
Title: How to generate labels for self-supervised training?
UserDisplayName: Vesko Vujovic
ThreadText: # How to generate labels for self-supervised training?
  
  (Asked by: username_0 on 2020-08-04)
  
  I've been reading a lot lately about self-supervised learning and I didn't understand very well how to generate the desired label for a given image.
  
  Let's say that I have an image classification task, and I have very little labeled data.
  
  ***How can I generate the target label from the other data in the dataset?***
  
  **Tags:** deep-learning, datasets, supervised-learning, self-supervised-learning, representation-learning
  > **Comments:**
  > **username_1**: Hi, I am also working on SSL task or more learning how to do it. Is it please possible, if you show me the way you did it please? :)  
  > **username_2**: @TimcoVanco See my [**updated** answer below](https://ai.stackexchange.com/a/22860/2444). It should clarify your doubts, at this point, I believe.  
  > **username_2**: You should accept my answer below (by clicking on the checkmark), if it answers your question. It should answer your question, at this point. If not, please, let me know what is still unclear.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-08-04. Score: 6
  
  > ***How can I generate the target label from the other data in the dataset?***
  
  If you are asking how you can create the learning signal in SSL, when given an unlabelled dataset, for learning representations of these unlabelled data, then there is **no general answer**. The answer depends on the type of data that you have (which can be e.g. textual or visual), and which features do you think you want to learn or can be learned from your unlabelled data. [This paper](https://arxiv.org/pdf/1902.06162.pdf) and other answers to [this question](https://ai.stackexchange.com/q/10623/2444) provide some examples of how that can be done (depending on the type of data). Below, I also provide an example.
  
  Let me try to explain this more in detail.
  
  Let's assume that you have *both*
  
  1. an **unlabelled** dataset $U = \{ u\_i \}\_{i=1}^m$ and
  2. a **labelled** dataset $D = \{(x\_i, y\_i) \}\_{i=1}^n$
  
  where we may have $m \gg n$ (although this is not a strict requirement), i.e. you may have a lot more unlabelled data than labelled data (this can easily be the case, given that, in general, manual data annotation is expensive/laborious). Let's say that your ultimate task is to perform object recognition (or classification). Let's call this task the **downstream task**. So, you may think that $x\_i$ and $u\_i$ are images and $y\_i$ are labels, like "cat" or "dog" (let's say that you want to differentiate between cats and dogs).
  
  You want to solve this downstream task by supervised learning with $D$. However, given that your labeled dataset is not big enough, you may think that training a neural network from scratch (i.e. by randomly initializing its weights) with $D$ may not lead to good performance. So, you think that it could be useful to start training from a pre-trained model that already contains useful representations of data similar to your labeled data, i.e. to perform [**transfer learning**](https://www.tensorflow.org/tutorials/images/transfer_learning). To pre-train such a model, you could use SSL.
  
  So, to solve your downstream task with SSL, there are **2 different steps**
  
  1. **Self-supervised learning (SSL)**: learn representations of your images $u\_i \in U$ by training a neural network $M$ with $U$ to solve a so-called **pretext** (or **auxiliary** task); there are many pre-text tasks: you can find many examples [here](https://arxiv.org/pdf/1902.06162.pdf), [here](https://ai.stackexchange.com/a/20393/2444) and [here](https://ai.stackexchange.com/q/10623/2444) (see example below too);
  2. **Supervised learning (SL)** by **transfer learning**: fine-tune $M$ with $D$ (the labeled dataset), in a supervised way; this task is known as **downstream task** (as stated above)
  
  In this process, there are **2 different labels**.
  
  * In step 1, you have the labels that are generated automatically. But how are these labels generated? As I said, there are many ways. Let me describe one way (among many others!). Let's say that your **unlabelled** dataset $U$ contains high-resolution images (i.e. $u\_i \in U$ are high-resolution images), then you could define your pre-text task as follows. You lower the resolution of your high-resolution images to create other images. Let $v\_i$ be the low-resolution image created from the high-resolution image $u\_i \in U$, then the training pair to your neural network $M$ is $(v\_i, u\_i) \in U'$, where $u\_i$ is the label (which is the original high-resolution image) and $U'$ the labeled dataset automatically generated (i.e. with the algorithm I've just explained).
  
    So, these labels $u\_i$ (high-resolution images) are semantically different than $y\_i$ ("cat" or "dog") in the pairs $(x\_i, y\_i) \in D$. They are different because, here, we want to learn representations and not to perform object recognition/classification: the idea is that, by solving this pre-text task, your final trained neural network, should have learned features of the images in the unlabelled data (i.e. representation learning). These learned features can then be used to bootstrap training in the downstream task.
  * In step 2, you use the labeled dataset $D$, which has been typically annotated (or labeled) by a human. As stated above, this dataset contains pairs $(x\_i, y\_i)$, where $y\_i$ is, for example, the label "cat" or "dog".
  
    In this step, the pre-trained model $M$, with the SSL technique, can be fine-tuned with $D$ in a supervised fashion. Given that we start with a pre-trained model $M$, we are effectively performing transfer learning.
  
  Note that SSL can also refer to something (slightly) different than what has been explained in this answer. See [my other answer](https://ai.stackexchange.com/a/10624/2444) for more details. Moreover, note that you can perform representation learning with SSL without necessarily solving a downstream task later, which may also not be an SL task (in the example above, I've described a downstream task that is an SL task only for simplicity).
  
  If this answer is still unclear, maybe you should have a look at existing implementations of SSL techniques (such as [this](https://github.com/google/revisiting-self-supervised)) for more inspiration.
  
  
  ---
  
  