Answers.Body: <p>BYOL is a self-supervised image representation learning method. The aim is to learn a representation <span class="math-container">$y_\theta$</span> of an image.</p>
  <blockquote>
  <p>Why is a prediction network added to the online network, which is not present in the target network?</p>
  </blockquote>
  <p>The target network has the same architecture of the online net (except the <em>predictor</em>) but its weights <span class="math-container">$\xi$</span> are a moving average of the online net. I think this is to avoid the need of building negative pairs for training the representation, conversely to prior approaches like <a href="http://proceedings.mlr.press/v119/chen20j/chen20j.pdf" rel="nofollow noreferrer">SimCLR</a>.</p>
  <ul>
  <li>Both projector and predictor share the same MLP architecture, that first expands and then compresses the embedding (representation) obtained by a Res-Net.</li>
  <li>As stated in table 5b and in the ablation, BYOL <em>without</em> the predictor achieves less performance. The authors also discovered that the predictor avoids <em>representation collapse</em>, and that also a near-optimal predictor removes the need for a target network.</li>
  </ul>
  <blockquote>
  <p>How are the online and target compared if they do not follow the same architecture?</p>
  </blockquote>
  <p>Since both projector and predictor have the same MLP architecture, you can train them by the loss fn in Eq. 2</p>
  <blockquote>
  <p>What 'prediction' is made by the prediction network (there are no labels?)?</p>
  </blockquote>
  <p>The whole BYOL is without labels, this is the point of SSL pre-training. The (self-)supervision is provided by the two different views <span class="math-container">$t$</span> and <span class="math-container">$t'$</span> of the input image. The views are obtained by applying two different augmentation functions.</p>
  <blockquote>
  <p>What is the intuition behind this section of the architecture?</p>
  </blockquote>
  <p>The BYOL training dynamics is compared to the ones of GAN models: see section 3.2. Since it does not explicitly use a term to prevent constant (i.e. collapsed) representations (as negative pairs, for example) a target network is necessary. Furthermore, the predictor introduces further asymmetry which helps learn better representations. A similar approach is also used in <a href="https://arxiv.org/abs/2011.10566" rel="nofollow noreferrer">SimSiam</a>, where the predictor is only used in one branch of the network and when paired with a projector is thought to be a sort of <em>encoder-decoder architecture</em> (see figure 1): the encoder is the projector, and the predictor the decoder.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T07:43:29.417Z
Answers.Text: I understand that it is SSL, but then still.. a 'predictor' network makes a prediction right? What exactly does it predict? I understand it doesn't predict labels. But it has to predict something if its called a predictor xD
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T08:22:31.320Z
Answers.Text: Your doubt is right, the name can be misleading. Actually, the "predictor" does not predict anything like a class label, or a regression value. It's just a further non-linear transformation (by an MLP) applied on top-of the projection, which in turn is applied to the representation (intermediate output of a ResNet). The predictor (as the projector) first expands to 4096, then ReLU,, and next shrinks to 256.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T08:34:51.493Z
Answers.Text: So what goal does the predictor have? What does it map towards?
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T08:43:43.617Z
Answers.Text: I've added my own answer which i found to help me more. It was a specific bit of information i was missing which i found in the recent cookbook publication. Ill give an upvote for your answer as well tho.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-11T18:24:06.737Z
Answers.LastActivityDate: 2023-05-11T18:24:06.737Z
Answers.UserDisplayName: Luca Anzalone
Answers.Body: <p>I've found the answer to my own confusion in <a href="https://arxiv.org/abs/2304.12210" rel="nofollow noreferrer">A Cookbook of Self-Supervised Learning</a>.</p>
  <blockquote>
  <p>&quot;BYOL uses two networks along with a predictor <strong>to map the outputs of one
  network to the other</strong>. The network predicting the output is called the online or student network while the network producing the target is called the target or teacher network. Each network receives a different view of the same image formed by image transformations including random resizing, cropping, color jittering, and brightness alterations. The student network is updated throughout training using gradient descent.&quot;</p>
  </blockquote>
  <p>To answer all the questions I originally had:</p>
  <ul>
  <li>Why is a prediction network added to the online network, which is not present in the target network?
  <ul>
  <li>To map the outputs of one network to the other.</li>
  </ul>
  </li>
  <li>How are the online and target compared if they do not follow the same architecture?
  <ul>
  <li>The prediction network maps the output of one network to the other. So they output in the 'same' space making it possible to be compared.</li>
  </ul>
  </li>
  <li>What 'prediction' is made by the prediction network (there are no labels?)?
  <ul>
  <li>It predicts the output of the other network.</li>
  </ul>
  </li>
  <li>What is the intuition behind this section of the architecture?
  <ul>
  <li>Read the above quote.</li>
  </ul>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-12T08:42:55.847Z
Answers.LastActivityDate: 2023-05-12T08:42:55.847Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>In the <a href="https://arxiv.org/abs/2006.07733" rel="nofollow noreferrer">BYOL paper</a>, the following architecture is presented:</p>
  <p><a href="https://i.sstatic.net/TGyke.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TGyke.png" alt="BYOL's architecture" /></a></p>
  <ul>
  <li>Why is a prediction network added to the online network, which is not present in the target network?</li>
  <li>How are the online and target compared if they do not follow the same architecture?</li>
  <li>What 'prediction' is made by the prediction network (there are no labels?)?</li>
  <li>What is the intuition behind this section of the architecture?</li>
  </ul>
  
Tags: <deep-learning><papers><architecture><representation-learning><self-supervised-learning>
Title: BYOL: Why is there a prediction network in the online network but not in the target network?
UserDisplayName: Robin van Hoorn
ThreadText: # BYOL: Why is there a prediction network in the online network but not in the target network?
  
  (Asked by: username_0 on 2023-05-11)
  
  In the [BYOL paper](https://arxiv.org/abs/2006.07733), the following architecture is presented:
  
  [![BYOL's architecture](https://i.sstatic.net/TGyke.png)](https://i.sstatic.net/TGyke.png)
  
  * Why is a prediction network added to the online network, which is not present in the target network?
  * How are the online and target compared if they do not follow the same architecture?
  * What 'prediction' is made by the prediction network (there are no labels?)?
  * What is the intuition behind this section of the architecture?
  
  **Tags:** deep-learning, papers, architecture, representation-learning, self-supervised-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-05-11. Score: 1
  
  BYOL is a self-supervised image representation learning method. The aim is to learn a representation $y\_\theta$ of an image.
  
  > Why is a prediction network added to the online network, which is not present in the target network?
  
  The target network has the same architecture of the online net (except the *predictor*) but its weights $\xi$ are a moving average of the online net. I think this is to avoid the need of building negative pairs for training the representation, conversely to prior approaches like [SimCLR](http://proceedings.mlr.press/v119/chen20j/chen20j.pdf).
  
  * Both projector and predictor share the same MLP architecture, that first expands and then compresses the embedding (representation) obtained by a Res-Net.
  * As stated in table 5b and in the ablation, BYOL *without* the predictor achieves less performance. The authors also discovered that the predictor avoids *representation collapse*, and that also a near-optimal predictor removes the need for a target network.
  
  > How are the online and target compared if they do not follow the same architecture?
  
  Since both projector and predictor have the same MLP architecture, you can train them by the loss fn in Eq. 2
  
  > What 'prediction' is made by the prediction network (there are no labels?)?
  
  The whole BYOL is without labels, this is the point of SSL pre-training. The (self-)supervision is provided by the two different views $t$ and $t'$ of the input image. The views are obtained by applying two different augmentation functions.
  
  > What is the intuition behind this section of the architecture?
  
  The BYOL training dynamics is compared to the ones of GAN models: see section 3.2. Since it does not explicitly use a term to prevent constant (i.e. collapsed) representations (as negative pairs, for example) a target network is necessary. Furthermore, the predictor introduces further asymmetry which helps learn better representations. A similar approach is also used in [SimSiam](https://arxiv.org/abs/2011.10566), where the predictor is only used in one branch of the network and when paired with a projector is thought to be a sort of *encoder-decoder architecture* (see figure 1): the encoder is the projector, and the predictor the decoder.
  
  > **Comments:**
  > **username_0**: I understand that it is SSL, but then still.. a 'predictor' network makes a prediction right? What exactly does it predict? I understand it doesn't predict labels. But it has to predict something if its called a predictor xD  
  > **username_1**: Your doubt is right, the name can be misleading. Actually, the "predictor" does not predict anything like a class label, or a regression value. It's just a further non-linear transformation (by an MLP) applied on top-of the projection, which in turn is applied to the representation (intermediate output of a ResNet). The predictor (as the projector) first expands to 4096, then ReLU,, and next shrinks to 256.  
  > **username_0**: So what goal does the predictor have? What does it map towards?  
  > **username_0**: I've added my own answer which i found to help me more. It was a specific bit of information i was missing which i found in the recent cookbook publication. Ill give an upvote for your answer as well tho.  
  
  ---
  
  ### Answer by username_0 on 2023-05-12. Score: 0
  
  I've found the answer to my own confusion in [A Cookbook of Self-Supervised Learning](https://arxiv.org/abs/2304.12210).
  
  > "BYOL uses two networks along with a predictor **to map the outputs of one
  > network to the other**. The network predicting the output is called the online or student network while the network producing the target is called the target or teacher network. Each network receives a different view of the same image formed by image transformations including random resizing, cropping, color jittering, and brightness alterations. The student network is updated throughout training using gradient descent."
  
  To answer all the questions I originally had:
  
  * Why is a prediction network added to the online network, which is not present in the target network?
    + To map the outputs of one network to the other.
  * How are the online and target compared if they do not follow the same architecture?
    + The prediction network maps the output of one network to the other. So they output in the 'same' space making it possible to be compared.
  * What 'prediction' is made by the prediction network (there are no labels?)?
    + It predicts the output of the other network.
  * What is the intuition behind this section of the architecture?
    + Read the above quote.
  
  
  ---
  
  