Answers.Body: <p>The main difference between distant supervision (as described in the link you provided) and self-supervision lies on the task the network is trained on.</p>
  <p>Distant supervision focuses on generating weak labels for the very same task that would be tackled with supervised labels, and the final result could be directly used for that matter.</p>
  <p>Self-supervision is a means for learning a data representation. It does so by learning a surrogate task, which is defined by inputs and labels derived exclusively from the original input data.</p>
  <p>I can imagine cases in which an implementation of self-supervision could be considered distant supervision (if the task casually matches the target task). On the other hand, if external data sources would be employed for training on a surrogate task, that would be a case of representation learning (that could incorporate self-supervision too).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T12:16:26.597Z
Answers.Text: Ok, taking https://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning/13749?noredirect=1#comment34138_13749 into account, the surrogate task is not the requirement, but learning from the input data is. While distant learning seems to be about learning from external data. I got that much. Now, the example of https://ai.stackexchange.com/a/10624/38174 would then *not* be self-supervised... unless you say that the proximity sensor is part of the "input", but then anything could be "part of the input".
Answers.UserDisplayName: Make42
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T12:18:17.473Z
Answers.Text: I would say the input must be the input of the target class, or not? But then, with a surrogate task, the "input" where the labels are extracted from, is *not* the input of the target class. E.g. the documents of word2vec would be the input where the labels are extracted from, but the input of a target task is the (one-hot) word-vector. So "input of target task" has its *origin* of the dataset from which labels are extracted, but it is not the same. Or am I missing something? In variational autoencoder they aren't the same either. In "normal" autoencoders however they are the same.
Answers.UserDisplayName: Make42
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T12:21:58.513Z
Answers.Text: Also, a lot of the publications mentioned in https://ai.stackexchange.com/questions/22184/does-self-supervised-learning-require-auxiliary-tasks claim that the auxiliary task the deciding factor. (The more I look into the naming in the machine learning literature, the more frustrating it gets.)
Answers.UserDisplayName: Make42
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T12:36:10.463Z
Answers.Text: 1) You got it right in the first comment. The robotics example is a bit farfetched, but it's correct as far as the sensor data is part of the input, but only trustworthy in some cases.
  2) The input is usually modified in order to generate the auxiliary task e.g. masking a word in order to predict it afterwards.
  3) The auxiliary task is key due to its implications in the representation learned e.g. predicting the mean color of an image is probably a very poor choice if you aim to perform facial recognition.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T15:12:34.983Z
Answers.Text: 1) When we say that "self-supervision requires that data labels are extracted from the input data", we need to define what the input data is. Look at the new graphic in the question: Is it the input to the target task (3), the input to the aux. task (2) or a data that lies even further "outside" (1). For word2vec it is (1). For variational autoencoders used for image denoising and my similar task we are actually extracting the (3) from the (1) and the off-the-shelf-data is the *out target* data. But: if (1) is the input data for the definition then *anything* could be part of that input data.
Answers.UserDisplayName: Make42
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T15:19:57.627Z
Answers.Text: 2) That is supporting my suspection that (1) is the input data. In that case the robotic case would also be self-supervised (if having an auxiliary task is no requirement), because we just say any data is input data in the (1)-sense. 3) The strong connection between representation learned and having an aux. task is obvious. However, if we want to define self-supervision and it is not supposed to be a requirement, we cannot use it in a (mathemathical) definition of self-supervision. And yes, I am searching for a definition that is good enough that it could be made into a math. def.
Answers.UserDisplayName: Make42
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T20:13:43.633Z
Answers.Text: @Make42 I really think you're overthinking it. Self-supervised learning is relatively new, and there's probably not a consensus on the definition of SSL. There's also the difference between SSL in robotics and in other contexts, as I emphasize in my answer. In the last edit of my answer, I attempt to give a more general definition of SSL: a machine learning technique that _automatically_ generates a supervisory signal (and I am emphasizing the word _automatically_).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-04T11:36:30.057Z
Answers.LastActivityDate: 2020-07-04T11:36:30.057Z
Answers.UserDisplayName: David
Body: <p><a href="https://en.wikipedia.org/wiki/Weak_supervision" rel="nofollow noreferrer">Weak supervision</a> is supervised learning, with uncertainty in the labeling, e.g. due to automatic labeling or because non-experts labelled the data [<a href="http://ai.stanford.edu/blog/weak-supervision/" rel="nofollow noreferrer">1</a>].</p>
  <p>Distant supervision [<a href="http://deepdive.stanford.edu/distant_supervision" rel="nofollow noreferrer">2</a>, <a href="https://stats.stackexchange.com/a/47036/83252">3</a>] is a type of weak supervision that uses an auxiliary automatic mechanism to produce weak labels / reference output (in contrast to non-expert human labelers).</p>
  <p>According to <a href="https://ai.stackexchange.com/a/10624/38174">this answer</a></p>
  <blockquote>
  <p>Self-supervised learning (or self-supervision) is a supervised learning technique where the training data is automatically labelled.</p>
  </blockquote>
  <p>In the examples for self-supervised learning, I have seen so far, the labels were extracted from the input data.</p>
  <p><strong>What is the difference between distant supervision and self-supervision?</strong></p>
  <ul>
  <li>Is it that for self-supervision, the labels must come from the input data and for distant supervision it <em>can</em> come from anywhere (which would make self-supervision a type of distant supervision)?</li>
  <li>Or <em>must</em> the labels from distant supervision come from somewhere else than the input data?</li>
  <li>If <a href="https://ai.stackexchange.com/a/10624/38174">&quot;In robotics, this can be done by finding and exploiting the relations or correlations between inputs coming from different sensor modalities.&quot;</a> then for self-supervised learning, the labels do not even have to originate from the input data. (Or did I misinterpret the quote?)</li>
  </ul>
  <hr />
  <p>(Setup mentioned in discussion:</p>
  <p><a href="https://i.sstatic.net/GOcRN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GOcRN.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-28T15:34:11.090Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109965/discussion-on-question-by-make42-what-is-the-difference-between-distant-supervis).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-04T18:03:06.130Z
Comments.Text: I have updated [my answer](https://ai.stackexchange.com/a/10624/2444). Anyway, after having read something about distant supervision, it seems like it refers to learning where the data comes from a remote database rather than a local dataset (and so the term "distant supervision" makes sense). In any case, this term is definitely less common than SSL.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-05T08:26:16.727Z
Comments.Text: @nbro: Which articles have you read regarding distant supervision? Maybe I can also have a look.
Comments.UserDisplayName: Make42
Tags: <comparison><supervised-learning><self-supervised-learning>
Title: What is the difference between distant supervision and self-supervision?
UserDisplayName: Make42
ThreadText: # What is the difference between distant supervision and self-supervision?
  
  (Asked by: username_0 on 2020-06-26)
  
  [Weak supervision](https://en.wikipedia.org/wiki/Weak_supervision) is supervised learning, with uncertainty in the labeling, e.g. due to automatic labeling or because non-experts labelled the data [[1](http://ai.stanford.edu/blog/weak-supervision/)].
  
  Distant supervision [[2](http://deepdive.stanford.edu/distant_supervision), [3](https://stats.stackexchange.com/a/47036/83252)] is a type of weak supervision that uses an auxiliary automatic mechanism to produce weak labels / reference output (in contrast to non-expert human labelers).
  
  According to [this answer](https://ai.stackexchange.com/a/10624/38174)
  
  > Self-supervised learning (or self-supervision) is a supervised learning technique where the training data is automatically labelled.
  
  In the examples for self-supervised learning, I have seen so far, the labels were extracted from the input data.
  
  **What is the difference between distant supervision and self-supervision?**
  
  * Is it that for self-supervision, the labels must come from the input data and for distant supervision it *can* come from anywhere (which would make self-supervision a type of distant supervision)?
  * Or *must* the labels from distant supervision come from somewhere else than the input data?
  * If ["In robotics, this can be done by finding and exploiting the relations or correlations between inputs coming from different sensor modalities."](https://ai.stackexchange.com/a/10624/38174) then for self-supervised learning, the labels do not even have to originate from the input data. (Or did I misinterpret the quote?)
  
  ---
  
  (Setup mentioned in discussion:
  
  [![enter image description here](https://i.sstatic.net/GOcRN.png)](https://i.sstatic.net/GOcRN.png)
  
  **Tags:** comparison, supervised-learning, self-supervised-learning
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109965/discussion-on-question-by-username_0-what-is-the-difference-between-distant-supervis).  
  > **username_1**: I have updated [my answer](https://ai.stackexchange.com/a/10624/2444). Anyway, after having read something about distant supervision, it seems like it refers to learning where the data comes from a remote database rather than a local dataset (and so the term "distant supervision" makes sense). In any case, this term is definitely less common than SSL.  
  > **username_0**: @username_1: Which articles have you read regarding distant supervision? Maybe I can also have a look.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-04. Score: 1
  
  The main difference between distant supervision (as described in the link you provided) and self-supervision lies on the task the network is trained on.
  
  Distant supervision focuses on generating weak labels for the very same task that would be tackled with supervised labels, and the final result could be directly used for that matter.
  
  Self-supervision is a means for learning a data representation. It does so by learning a surrogate task, which is defined by inputs and labels derived exclusively from the original input data.
  
  I can imagine cases in which an implementation of self-supervision could be considered distant supervision (if the task casually matches the target task). On the other hand, if external data sources would be employed for training on a surrogate task, that would be a case of representation learning (that could incorporate self-supervision too).
  
  > **Comments:**
  > **username_0**: Ok, taking https://ai.stackexchange.com/questions/10623/what-is-self-supervised-learning-in-machine-learning/13749?noredirect=1#comment34138_13749 into account, the surrogate task is not the requirement, but learning from the input data is. While distant learning seems to be about learning from external data. I got that much. Now, the example of https://ai.stackexchange.com/a/10624/38174 would then *not* be self-supervised... unless you say that the proximity sensor is part of the "input", but then anything could be "part of the input".  
  > **username_0**: I would say the input must be the input of the target class, or not? But then, with a surrogate task, the "input" where the labels are extracted from, is *not* the input of the target class. E.g. the documents of word2vec would be the input where the labels are extracted from, but the input of a target task is the (one-hot) word-vector. So "input of target task" has its *origin* of the dataset from which labels are extracted, but it is not the same. Or am I missing something? In variational autoencoder they aren't the same either. In "normal" autoencoders however they are the same.  
  > **username_0**: Also, a lot of the publications mentioned in https://ai.stackexchange.com/questions/22184/does-self-supervised-learning-require-auxiliary-tasks claim that the auxiliary task the deciding factor. (The more I look into the naming in the machine learning literature, the more frustrating it gets.)  
  > **username_2**: 1) You got it right in the first comment. The robotics example is a bit farfetched, but it's correct as far as the sensor data is part of the input, but only trustworthy in some cases. 2) The input is usually modified in order to generate the auxiliary task e.g. masking a word in order to predict it afterwards. 3) The auxiliary task is key due to its implications in the representation learned e.g. predicting the mean color of an image is probably a very poor choice if you aim to perform facial recognition.  
  > **username_0**: 1) When we say that "self-supervision requires that data labels are extracted from the input data", we need to define what the input data is. Look at the new graphic in the question: Is it the input to the target task (3), the input to the aux. task (2) or a data that lies even further "outside" (1). For word2vec it is (1). For variational autoencoders used for image denoising and my similar task we are actually extracting the (3) from the (1) and the off-the-shelf-data is the *out target* data. But: if (1) is the input data for the definition then *anything* could be part of that input data.  
  > **username_0**: 2) That is supporting my suspection that (1) is the input data. In that case the robotic case would also be self-supervised (if having an auxiliary task is no requirement), because we just say any data is input data in the (1)-sense. 3) The strong connection between representation learned and having an aux. task is obvious. However, if we want to define self-supervision and it is not supposed to be a requirement, we cannot use it in a (mathemathical) definition of self-supervision. And yes, I am searching for a definition that is good enough that it could be made into a math. def.  
  > **username_1**: @username_0 I really think you're overthinking it. Self-supervised learning is relatively new, and there's probably not a consensus on the definition of SSL. There's also the difference between SSL in robotics and in other contexts, as I emphasize in my answer. In the last edit of my answer, I attempt to give a more general definition of SSL: a machine learning technique that _automatically_ generates a supervisory signal (and I am emphasizing the word _automatically_).  
  
  ---
  
  