Answers.Body: <p>This is exactly the problem I am currently working on. I don't suggest that you use a supervised method to learn latent representation, as the model might <a href="https://arxiv.org/abs/2004.07780" rel="nofollow noreferrer">learn shortcuts</a> or only a most meaningful feature ignoring other latent features.</p>
  <p>There are several self-supervised representation learning approaches:</p>
  <ol>
  <li><a href="https://arxiv.org/abs/2002.05709" rel="nofollow noreferrer">SimCLR</a> and <a href="https://arxiv.org/abs/2006.10029" rel="nofollow noreferrer">SimCLR2</a></li>
  <li><a href="https://arxiv.org/abs/2006.09882" rel="nofollow noreferrer">SwAV</a></li>
  <li><a href="https://arxiv.org/abs/2011.10566" rel="nofollow noreferrer">SimSiam</a></li>
  </ol>
  <p>Here is an <a href="https://lilianweng.github.io/posts/2019-11-10-self-supervised/" rel="nofollow noreferrer">overview</a> of such approaches.</p>
  <p>These approaches are aimed to learn a latent representation of images (pre-text task) that can be then used for any downstream tasks (classification, query similar samples). Here is an interesting <a href="https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/" rel="nofollow noreferrer">article</a> on this topic.</p>
  <p>Another approach is to use generative models. The benefit of this approach is that you can also generate synthetic images from the embeddings that can help debug and evaluate your model.</p>
  <p>One of the simplest architectures is autoencoders (VAE, VQ-VAE). Another option is generative adversarial networks. They are more difficult to train but can produce higher quality images than autoencoders. Another advantage is that the latent space is more disentangled meaning that each value might be semantically interpreted.</p>
  <p>The main limitation is that the conventional GAN can produce samples only from random noise and to encode a real image the model has to be extended. There are different approaches to overcome this limitation, for example <a href="https://arxiv.org/abs/2004.04467" rel="nofollow noreferrer">ALAE</a>.</p>
  <p>This <a href="https://openreview.net/pdf?id=NeRdBeTionN" rel="nofollow noreferrer">paper</a> compares some of these approaches.</p>
  <p>While these approaches can easily capture coarse details such as hairstyle or background color, they are likely to ignore finer details. For medical imaging, the main goal is to identify cancer cells, which is essential for diagnosis. If the model does not learn this feature, it renders a latent representation of no avail.</p>
  <p>However, latent space can be forced to take these features into account by guiding the model with a supervised downstream task such as classification (<a href="https://arxiv.org/abs/2104.13369" rel="nofollow noreferrer">StylEX</a>) or segmentation (<a href="https://arxiv.org/pdf/2111.03186.pdf" rel="nofollow noreferrer">EditGAN</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-13T14:42:19.373Z
Answers.Text: Thank you for the thorough comment! I'll accept it as the answer!
Answers.UserDisplayName: Adrien Nivaggioli
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-13T10:38:55.270Z
Answers.LastActivityDate: 2022-05-13T10:38:55.270Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>I'd like to build a deep feature extractor of images (using a Bi-linear CNN).</p>
  <p>What would lead to the best results:</p>
  <ul>
  <li>an unsupervised approach (such as <a href="https://iopscience.iop.org/article/10.1088/1742-6596/1237/3/032044/meta" rel="nofollow noreferrer">https://iopscience.iop.org/article/10.1088/1742-6596/1237/3/032044/meta</a>), or</li>
  <li>training a classifier on as many classes as I can, and hope it generalized enough?</li>
  </ul>
  <p>I would then like to use this extractor as:</p>
  <ul>
  <li>A weights initialization for other classical tasks</li>
  <li>A feature extractor for Few Shot Learning approaches</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-12T07:46:19.807Z
Comments.Text: Depends on what your downstream task is. There are so many ways to pretrain image feature extractors, and you will have to choose them specifically based on their performance on you desired downstream task.
Comments.UserDisplayName: DKDK
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-13T09:09:44.357Z
Comments.Text: Please, edit your post to provide the details that are missing, which are mentioned in the previous task, i.e. what is/are your downstream task(s)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-13T09:21:01.517Z
Comments.Text: Thank you for your answers, I've done so!
Comments.UserDisplayName: Adrien Nivaggioli
Tags: <deep-learning><deep-neural-networks><unsupervised-learning>
Title: Should I use an unsupervised approach or train a classifier with many classes to build a deep image feature extractor?
UserDisplayName: Adrien Nivaggioli
ThreadText: # Should I use an unsupervised approach or train a classifier with many classes to build a deep image feature extractor?
  
  (Asked by: username_0 on 2022-05-10)
  
  I'd like to build a deep feature extractor of images (using a Bi-linear CNN).
  
  What would lead to the best results:
  
  * an unsupervised approach (such as <https://iopscience.iop.org/article/10.1088/1742-6596/1237/3/032044/meta>), or
  * training a classifier on as many classes as I can, and hope it generalized enough?
  
  I would then like to use this extractor as:
  
  * A weights initialization for other classical tasks
  * A feature extractor for Few Shot Learning approaches
  
  **Tags:** deep-learning, deep-neural-networks, unsupervised-learning
  
  > **Comments:**
  > **username_1**: Depends on what your downstream task is. There are so many ways to pretrain image feature extractors, and you will have to choose them specifically based on their performance on you desired downstream task.  
  > **username_2**: Please, edit your post to provide the details that are missing, which are mentioned in the previous task, i.e. what is/are your downstream task(s)?  
  > **username_0**: Thank you for your answers, I've done so!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-05-13. Score: 1 | Accepted answer
  
  This is exactly the problem I am currently working on. I don't suggest that you use a supervised method to learn latent representation, as the model might [learn shortcuts](https://arxiv.org/abs/2004.07780) or only a most meaningful feature ignoring other latent features.
  
  There are several self-supervised representation learning approaches:
  
  1. [SimCLR](https://arxiv.org/abs/2002.05709) and [SimCLR2](https://arxiv.org/abs/2006.10029)
  2. [SwAV](https://arxiv.org/abs/2006.09882)
  3. [SimSiam](https://arxiv.org/abs/2011.10566)
  
  Here is an [overview](https://lilianweng.github.io/posts/2019-11-10-self-supervised/) of such approaches.
  
  These approaches are aimed to learn a latent representation of images (pre-text task) that can be then used for any downstream tasks (classification, query similar samples). Here is an interesting [article](https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/) on this topic.
  
  Another approach is to use generative models. The benefit of this approach is that you can also generate synthetic images from the embeddings that can help debug and evaluate your model.
  
  One of the simplest architectures is autoencoders (VAE, VQ-VAE). Another option is generative adversarial networks. They are more difficult to train but can produce higher quality images than autoencoders. Another advantage is that the latent space is more disentangled meaning that each value might be semantically interpreted.
  
  The main limitation is that the conventional GAN can produce samples only from random noise and to encode a real image the model has to be extended. There are different approaches to overcome this limitation, for example [ALAE](https://arxiv.org/abs/2004.04467).
  
  This [paper](https://openreview.net/pdf?id=NeRdBeTionN) compares some of these approaches.
  
  While these approaches can easily capture coarse details such as hairstyle or background color, they are likely to ignore finer details. For medical imaging, the main goal is to identify cancer cells, which is essential for diagnosis. If the model does not learn this feature, it renders a latent representation of no avail.
  
  However, latent space can be forced to take these features into account by guiding the model with a supervised downstream task such as classification ([StylEX](https://arxiv.org/abs/2104.13369)) or segmentation ([EditGAN](https://arxiv.org/pdf/2111.03186.pdf)).
  
  > **Comments:**
  > **username_0**: Thank you for the thorough comment! I'll accept it as the answer!  
  
  ---
  
  