Answers.Body: <p>Both semi-supervised and self-supervised methods are similar in the sense that the goal is to learn with fewer labels per class. The way both formulate this is quite different:  </p>
  
  <ol>
  <li><strong>Self-Supervised Learning:</strong>    </li>
  </ol>
  
  <p>This line of work aims to learn image representations without requiring human-annotated labels and then use those learned representations on some downstream tasks. For example, you could take millions of unlabeled images, randomly rotate them by either 0, 90, 180 or 270 degrees and then train a model to predict the rotation angle. Once the model is trained, you can use transfer learning to fine-tune this model on a downstream task like cat/dog classification just like how you finetune ImageNet pretrained models. You can view an <a href="https://amitness.com/2020/02/illustrated-self-supervised-learning/" rel="noreferrer"><strong>overview</strong></a> of the methods and also look at contrastive learning methods that are currently giving state-of-the-art results such as <a href="https://amitness.com/2020/03/illustrated-simclr/" rel="noreferrer"><strong>SimCLR</strong></a> and <a href="https://amitness.com/2020/03/illustrated-pirl/" rel="noreferrer"><strong>PIRL</strong></a>.</p>
  
  <p><img src="https://amitness.com/images/ss-geometric-transformation-architecture.png" alt=""></p>
  
  <ol start="2">
  <li><strong>Semi-supervised Learning</strong>  </li>
  </ol>
  
  <p>Different from self-supervised learning, semi-supervised learning aims to use both labeled and unlabeled data at the same time to improve the performance of a supervised model. An example of this is <a href="https://amitness.com/2020/03/fixmatch-semi-supervised/" rel="noreferrer">FixMatch</a> paper where you train your model on labeled images. Then, for your unlabeled images, you apply augmentations to create two images for each unlabeled image. Now, we want to ensure that the model predicts the same label for both the augmentations of the unlabeled images. This can be incorporated into the loss as a cross-entropy loss. </p>
  
  <p><img src="https://amitness.com/images/fixmatch-unlabeled-augment-concept.png" alt=""></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-19T10:48:23.937Z
Answers.LastActivityDate: 2020-04-19T10:48:23.937Z
Answers.UserDisplayName: Amit Chaudhary
Answers.Body: <p>The previous answer has given a good insight into the difference between two areas. I would like to give more examples. </p>
  
  <p><strong>Semi-Supervised Learning</strong> work with improving the data set by adding up new examples. There are iterative systems where we train a model on a given dataset and improve the model further after deploying it on the real world by adding interactions of the real world and their outcomes to further train the system.</p>
  
  <p><strong>Self-Supervised Learning</strong> is becoming a very hot topic these days. It has the ability to understand the underline properties of a given dataset with some kind of a supervisory signal (Not exactly a label). <strong>self-Attention</strong> introduced in Transformers is a modern day popular self-supervised learning. Also, check this tweet from <a href="https://twitter.com/ylecun/status/1123235709802905600?lang=en" rel="noreferrer">Yann Lecun tweet</a>   </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-03T00:05:55.163Z
Answers.LastActivityDate: 2019-07-03T00:05:55.163Z
Answers.UserDisplayName: Shamane Siriwardhana
Answers.Body: <h3>Semi-supervised learning</h3>
  <p><a href="https://en.wikipedia.org/wiki/Semi-supervised_learning" rel="nofollow noreferrer">Semi-supervised learning</a> is the collection of machine learning techniques where there are two datasets: a labelled one and an unlabelled one.</p>
  <p>There are two main problems that can be solved using semi-supervised learning:</p>
  <ul>
  <li>transductive learning (i.e. label the given unlabelled data) and</li>
  <li>inductive learning (generalization) (i.e. find a function that maps inputs to outputs, like classification).</li>
  </ul>
  <h3>Self-supervised learning</h3>
  <p><a href="https://ai.stackexchange.com/a/10624/2444">Self-supervised learning</a> (SSL) is a machine learning approach where the supervisory signal is <strong>automatically</strong> generated. More precisely, SSL can either refer to</p>
  <ul>
  <li><p>learn data representations (i.e. learn to represent the data) by solving a so-called <strong>pretext</strong> (or auxiliary) task, in a self-supervised fashion, i.e. you automatically generate the supervised signal from the unlabelled</p>
  </li>
  <li><p>automatically label an unlabelled dataset by exploiting data coming from different sensors (this is the usual definition of SLL in the context or robotics)</p>
  </li>
  </ul>
  <h3>What is the relationship between the two?</h3>
  <p>SSL (for data representation) can be considered a semi-supervised learning approach, if you fine-tune the learned data representations with a labeled dataset to solve supervised learning problem (i.e. the so-called <strong>downstream</strong> task), which you will probably do otherwise the data representations are pretty useless. Read <a href="https://ai.stackexchange.com/a/22860/2444">this answer</a> for other details.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-12T01:16:51.387Z
Answers.LastActivityDate: 2020-08-04T19:02:26.430Z
Answers.LastEditDate: 2020-08-04T19:02:26.430Z
Answers.UserDisplayName: nbro
Body: <p>What's the differences between <a href="https://www.scihive.org/paper/1905.03670" rel="noreferrer">semi-supervised learning and self-supervised visual representation learning</a>, and how they are connected? </p>
  
Tags: <machine-learning><comparison><supervised-learning><self-supervised-learning><semi-supervised-learning>
Title: What is the relation between semi-supervised and self-supervised visual representation learning?
UserDisplayName: 0x90
ThreadText: # What is the relation between semi-supervised and self-supervised visual representation learning?
  
  (Asked by: username_0 on 2019-05-12)
  
  What's the differences between [semi-supervised learning and self-supervised visual representation learning](https://www.scihive.org/paper/1905.03670), and how they are connected?
  
  **Tags:** machine-learning, comparison, supervised-learning, self-supervised-learning, semi-supervised-learning
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-04-19. Score: 11
  
  Both semi-supervised and self-supervised methods are similar in the sense that the goal is to learn with fewer labels per class. The way both formulate this is quite different:
  
  1. **Self-Supervised Learning:**
  
  This line of work aims to learn image representations without requiring human-annotated labels and then use those learned representations on some downstream tasks. For example, you could take millions of unlabeled images, randomly rotate them by either 0, 90, 180 or 270 degrees and then train a model to predict the rotation angle. Once the model is trained, you can use transfer learning to fine-tune this model on a downstream task like cat/dog classification just like how you finetune ImageNet pretrained models. You can view an [**overview**](https://amitness.com/2020/02/illustrated-self-supervised-learning/) of the methods and also look at contrastive learning methods that are currently giving state-of-the-art results such as [**SimCLR**](https://amitness.com/2020/03/illustrated-simclr/) and [**PIRL**](https://amitness.com/2020/03/illustrated-pirl/).
  
  ![](https://amitness.com/images/ss-geometric-transformation-architecture.png)
  
  2. **Semi-supervised Learning**
  
  Different from self-supervised learning, semi-supervised learning aims to use both labeled and unlabeled data at the same time to improve the performance of a supervised model. An example of this is [FixMatch](https://amitness.com/2020/03/fixmatch-semi-supervised/) paper where you train your model on labeled images. Then, for your unlabeled images, you apply augmentations to create two images for each unlabeled image. Now, we want to ensure that the model predicts the same label for both the augmentations of the unlabeled images. This can be incorporated into the loss as a cross-entropy loss.
  
  ![](https://amitness.com/images/fixmatch-unlabeled-augment-concept.png)
  
  
  ---
  
  ### Answer by username_3 on 2019-05-12. Score: 8
  
  ### Semi-supervised learning
  
  [Semi-supervised learning](https://en.wikipedia.org/wiki/Semi-supervised_learning) is the collection of machine learning techniques where there are two datasets: a labelled one and an unlabelled one.
  
  There are two main problems that can be solved using semi-supervised learning:
  
  * transductive learning (i.e. label the given unlabelled data) and
  * inductive learning (generalization) (i.e. find a function that maps inputs to outputs, like classification).
  
  ### Self-supervised learning
  
  [Self-supervised learning](https://ai.stackexchange.com/a/10624/2444) (SSL) is a machine learning approach where the supervisory signal is **automatically** generated. More precisely, SSL can either refer to
  
  * learn data representations (i.e. learn to represent the data) by solving a so-called **pretext** (or auxiliary) task, in a self-supervised fashion, i.e. you automatically generate the supervised signal from the unlabelled
  * automatically label an unlabelled dataset by exploiting data coming from different sensors (this is the usual definition of SLL in the context or robotics)
  
  ### What is the relationship between the two?
  
  SSL (for data representation) can be considered a semi-supervised learning approach, if you fine-tune the learned data representations with a labeled dataset to solve supervised learning problem (i.e. the so-called **downstream** task), which you will probably do otherwise the data representations are pretty useless. Read [this answer](https://ai.stackexchange.com/a/22860/2444) for other details.
  
  
  ---
  
  ### Answer by username_2 on 2019-07-03. Score: 5
  
  The previous answer has given a good insight into the difference between two areas. I would like to give more examples.
  
  **Semi-Supervised Learning** work with improving the data set by adding up new examples. There are iterative systems where we train a model on a given dataset and improve the model further after deploying it on the real world by adding interactions of the real world and their outcomes to further train the system.
  
  **Self-Supervised Learning** is becoming a very hot topic these days. It has the ability to understand the underline properties of a given dataset with some kind of a supervisory signal (Not exactly a label). **self-Attention** introduced in Transformers is a modern day popular self-supervised learning. Also, check this tweet from [Yann Lecun tweet](https://twitter.com/ylecun/status/1123235709802905600?lang=en)
  
  
  ---
  
  