Answers.Body: <p>Most (if not all) self-supervised learning techniques for (visual or textual) representation learning use pre-text tasks, and many pre-text tasks have been proposed in recent years.</p>
  <p>However, as I say in <a href="https://ai.stackexchange.com/a/10624/2444">my other answer</a> (which you cite), the term SSL has <strong>also</strong> been used (at least, in robotics: for example, see <a href="https://arxiv.org/pdf/1809.07207.pdf" rel="nofollow noreferrer">this paper</a>, which I am very familiar with) to refer to techniques that automatically (although approximately) label the unlabelled dataset for your downstream task (i.e. image recognition), i.e. they automatically create a labeled dataset of pairs <span class="math-container">$(x_i, \hat{y}_i)$</span>, where <span class="math-container">$x_i$</span> is an image that contains an object and <span class="math-container">$\hat{y}_i$</span> is the automatically (and possibly approximately) generated label (such as &quot;dog&quot;). This latter use of the term SSL is closer to some <a href="https://watermark.silverchair.com/nwx106.pdf" rel="nofollow noreferrer">weakly supervised learning (WSL)</a> techniques. Actually, it can be considered a WSL technique.</p>
  <p>Now, in <a href="https://arxiv.org/pdf/1809.07207.pdf" rel="nofollow noreferrer">this specific paper</a>, they actually solve some kind of pre-text task, i.e. they exploit the relations between two different sensors to produce the labels.</p>
  <p>To answer your question more directly: in all SSL papers that I have come across, some kind of pre-text task is always solved, i.e., in some way, you need to automatically generate the supervisory signal, and that task that we solve with the automatically generated learning signal (with the purpose of learning representations or generating a labeled dataset) can be considered the pre-text task (which may coincide with the downstream task, for example, in the case you're training an auto-encoder with an unlabelled dataset for the purposes of image denoising).</p>
  <p>In any case, I wouldn't bother too much about it. Just keep your context in mind when reading your paper. If you're really worried about it, then you should probably read almost all SSL-related papers, but, in that case, by the end of that, you will be an expert on the topic and you will not need our help (or my help).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-20T17:15:04.077Z
Answers.LastActivityDate: 2020-11-21T20:07:58.807Z
Answers.LastEditDate: 2020-11-21T20:07:58.807Z
Answers.UserDisplayName: nbro
Body: <p>Self-supervised learning algorithms provide labels automatically. But, it is not clear what else is required for an algorithm to fall under the category &quot;self-supervised&quot;:</p>
  <p>Some say, self-supervised learning algorithms learn on a set of <em>auxiliary tasks</em> [<a href="https://stats.stackexchange.com/questions/434224/how-to-differentiate-auto-encoder-techniques-from-self-supervised-learning">1</a>], also named <em>pretext task</em> [<a href="https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html" rel="nofollow noreferrer">2</a>, <a href="https://ai.stackexchange.com/a/10743/38174">3</a>], instead of the task we are interested in. Further examples are word2vec or autoencoders [<a href="https://link.springer.com/chapter/10.1007/978-3-319-46672-9_10" rel="nofollow noreferrer">4</a>] or word2vec [<a href="https://hackernoon.com/self-supervised-learning-gets-us-closer-to-autonomous-learning-be77e6c86b5a" rel="nofollow noreferrer">5</a>]. Here it is sometimes mentioned that the goal is to &quot;expose the inner structure of the data&quot;.</p>
  <p>Others do not mention that, implying that some algorithms can be called to be &quot;self-supervised learning algorithms&quot; if they are directly learning the task we are interested in [<a href="https://ai.stackexchange.com/a/10624/38174">6</a>, <a href="https://medium.com/@behnamsabeti/various-types-of-supervision-in-machine-learning-c7f32c190fbe" rel="nofollow noreferrer">7</a>].</p>
  <p><strong>Is the &quot;auxiliary tasks&quot; a requirement for a training setup to be called &quot;self-supervised learning&quot; or is it just optional?</strong></p>
  <hr />
  <h2>Research articles mentioning the auxiliary / pretext task:</h2>
  <ol>
  <li><a href="https://arxiv.org/abs/1901.09005" rel="nofollow noreferrer">Revisiting Self-Supervised Visual Representation Learning, 2019</a>, mentioned by [<a href="https://ai.stackexchange.com/a/10743/38174">3</a>]:</li>
  </ol>
  <blockquote>
  <p>The self-supervised learning framework requires only unlabeled data in order to formulate a pretext learning task such as predicting context or image rotation, for which a target objective can be computed without supervision.</p>
  </blockquote>
  <ol start="2">
  <li><a href="https://arxiv.org/abs/1803.07728" rel="nofollow noreferrer">Unsupervised Representation Learning by Predicting Image Rotations,  ICLR, 2018</a>, mentioned by
  [<a href="https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html" rel="nofollow noreferrer">2</a>]:</li>
  </ol>
  <blockquote>
  <p>a prominent paradigm is the so-called self-supervised learning that defines an annotation free pretext task, using only the visual information present on the images or videos, in order to provide a surrogatesupervision signal for feature learning.</p>
  </blockquote>
  <ol start="3">
  <li><a href="https://arxiv.org/abs/1505.05192" rel="nofollow noreferrer">Unsupervised Visual Representation Learning by Context Prediction, 2016</a>, mentioned by
  [<a href="https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html" rel="nofollow noreferrer">2</a>]:</li>
  </ol>
  <blockquote>
  <p>This converts an apparently unsupervised problem (finding a good similarity metric between words) intoa “self-supervised” one:  learning a function from a givenword to the words surrounding it.  Here the context predic-tion  task  is  just  a  “pretext”  to  force  the  model  to  learn  agood word embedding, which, in turn, has been shown tobe useful in a number of real tasks, such as semantic wordsimilarity.</p>
  </blockquote>
  <ol start="4">
  <li><a href="https://arxiv.org/abs/1905.01235" rel="nofollow noreferrer">Scaling and Benchmarking Self-Supervised Visual Representation Learning, 2019</a>:</li>
  </ol>
  <blockquote>
  <p>In discriminative self-supervised learning, which is the main focus of this work, a model is trained on an auxiliary or ‘pretext’ task for which ground-truth is available for free. In most cases, the pretext task involves predicting some hidden portion of the data (for example, predicting color for gray-scale images</p>
  </blockquote>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-28T15:33:15.590Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109964/discussion-on-question-by-make42-does-self-supervised-learning-require-auxiliary).
Comments.UserDisplayName: nbro
Tags: <definitions><transfer-learning><representation-learning><self-supervised-learning><pretext-tasks>
Title: Does self-supervised learning require auxiliary tasks?
UserDisplayName: Make42
ThreadText: # Does self-supervised learning require auxiliary tasks?
  
  (Asked by: username_0 on 2020-06-26)
  
  Self-supervised learning algorithms provide labels automatically. But, it is not clear what else is required for an algorithm to fall under the category "self-supervised":
  
  Some say, self-supervised learning algorithms learn on a set of *auxiliary tasks* [[1](https://stats.stackexchange.com/questions/434224/how-to-differentiate-auto-encoder-techniques-from-self-supervised-learning)], also named *pretext task* [[2](https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html), [3](https://ai.stackexchange.com/a/10743/38174)], instead of the task we are interested in. Further examples are word2vec or autoencoders [[4](https://link.springer.com/chapter/10.1007/978-3-319-46672-9_10)] or word2vec [[5](https://hackernoon.com/self-supervised-learning-gets-us-closer-to-autonomous-learning-be77e6c86b5a)]. Here it is sometimes mentioned that the goal is to "expose the inner structure of the data".
  
  Others do not mention that, implying that some algorithms can be called to be "self-supervised learning algorithms" if they are directly learning the task we are interested in [[6](https://ai.stackexchange.com/a/10624/38174), [7](https://medium.com/@behnamsabeti/various-types-of-supervision-in-machine-learning-c7f32c190fbe)].
  
  **Is the "auxiliary tasks" a requirement for a training setup to be called "self-supervised learning" or is it just optional?**
  
  ---
  
  ## Research articles mentioning the auxiliary / pretext task:
  
  1. [Revisiting Self-Supervised Visual Representation Learning, 2019](https://arxiv.org/abs/1901.09005), mentioned by [[3](https://ai.stackexchange.com/a/10743/38174)]:
  
  > The self-supervised learning framework requires only unlabeled data in order to formulate a pretext learning task such as predicting context or image rotation, for which a target objective can be computed without supervision.
  
  2. [Unsupervised Representation Learning by Predicting Image Rotations, ICLR, 2018](https://arxiv.org/abs/1803.07728), mentioned by
     [[2](https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html)]:
  
  > a prominent paradigm is the so-called self-supervised learning that defines an annotation free pretext task, using only the visual information present on the images or videos, in order to provide a surrogatesupervision signal for feature learning.
  
  3. [Unsupervised Visual Representation Learning by Context Prediction, 2016](https://arxiv.org/abs/1505.05192), mentioned by
     [[2](https://lilianweng.github.io/lil-log/2019/11/10/self-supervised-learning.html)]:
  
  > This converts an apparently unsupervised problem (finding a good similarity metric between words) intoa “self-supervised” one: learning a function from a givenword to the words surrounding it. Here the context predic-tion task is just a “pretext” to force the model to learn agood word embedding, which, in turn, has been shown tobe useful in a number of real tasks, such as semantic wordsimilarity.
  
  4. [Scaling and Benchmarking Self-Supervised Visual Representation Learning, 2019](https://arxiv.org/abs/1905.01235):
  
  > In discriminative self-supervised learning, which is the main focus of this work, a model is trained on an auxiliary or ‘pretext’ task for which ground-truth is available for free. In most cases, the pretext task involves predicting some hidden portion of the data (for example, predicting color for gray-scale images
  
  **Tags:** definitions, transfer-learning, representation-learning, self-supervised-learning, pretext-tasks
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/109964/discussion-on-question-by-make42-does-self-supervised-learning-require-auxiliary).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-20. Score: 0
  
  Most (if not all) self-supervised learning techniques for (visual or textual) representation learning use pre-text tasks, and many pre-text tasks have been proposed in recent years.
  
  However, as I say in [my other answer](https://ai.stackexchange.com/a/10624/2444) (which you cite), the term SSL has **also** been used (at least, in robotics: for example, see [this paper](https://arxiv.org/pdf/1809.07207.pdf), which I am very familiar with) to refer to techniques that automatically (although approximately) label the unlabelled dataset for your downstream task (i.e. image recognition), i.e. they automatically create a labeled dataset of pairs $(x\_i, \hat{y}\_i)$, where $x\_i$ is an image that contains an object and $\hat{y}\_i$ is the automatically (and possibly approximately) generated label (such as "dog"). This latter use of the term SSL is closer to some [weakly supervised learning (WSL)](https://watermark.silverchair.com/nwx106.pdf) techniques. Actually, it can be considered a WSL technique.
  
  Now, in [this specific paper](https://arxiv.org/pdf/1809.07207.pdf), they actually solve some kind of pre-text task, i.e. they exploit the relations between two different sensors to produce the labels.
  
  To answer your question more directly: in all SSL papers that I have come across, some kind of pre-text task is always solved, i.e., in some way, you need to automatically generate the supervisory signal, and that task that we solve with the automatically generated learning signal (with the purpose of learning representations or generating a labeled dataset) can be considered the pre-text task (which may coincide with the downstream task, for example, in the case you're training an auto-encoder with an unlabelled dataset for the purposes of image denoising).
  
  In any case, I wouldn't bother too much about it. Just keep your context in mind when reading your paper. If you're really worried about it, then you should probably read almost all SSL-related papers, but, in that case, by the end of that, you will be an expert on the topic and you will not need our help (or my help).
  
  
  ---
  
  