Answers.Body: <p>During pre-training, BERT learns to associate the [CLS] token’s hidden state with the semantic content of the entire input, and the classifier on top of the [CLS] token's representation helps BERT perform various downstream tasks like sentence classification.</p>
  <p>The distillation process focuses on aligning the hidden states of all tokens, including the [CLS] token via the required distillation loss which ensures that DistilBERT's representation of the [CLS] token is similar to the [CLS] token's representation in BERT. Also unlike the MLM loss, the additional optional cosine loss recommended by the original paper likely also focuses on aligning the hidden states of all tokens. Finally as you rightly mentioned, DistilBERT starts with BERT's weights initially, so the hidden state of the [CLS] token in DistilBERT is already in a good starting position to represent the entire input sequence in a similar manner to BERT. The distillation pretraining further refines and mimics this representation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-08T23:24:07.987Z
Answers.LastActivityDate: 2024-12-08T23:24:07.987Z
Answers.UserDisplayName: cinch
Body: <p>BERT's pre-training involves next sentence prediction (NSP) based on a classifier on top of the [CLS] token's last hidden state. This is primarily what gives the [CLS] token's hidden state the special ability to represent the entire input, yes?</p>
  <p>DistilBERT's pre-training does not involve NSP. How can then the [CLS] token's hidden state from DistilBERT represent the entire input? I understand that DistilBERT's weights are initialized with corresponding weights from BERT, but is that enough?</p>
  <p>If the triple loss used in DistilBERT's distillation process aligns hidden states of <em>all</em> tokens with corresponding states from BERT, I can see how [CLS] token's hidden state would also be aligned. But <a href="https://arxiv.org/abs/1910.01108" rel="nofollow noreferrer">the original DistilBERT paper</a> seems to focus only on the masked tokens in the loss function, and the [CLS] token is not masked.</p>
  
Tags: <bert><loss>
Title: How does DistilBERT's training ensure that the [CLS] token's hidden state from DistilBERT aligns with that from BERT?
UserDisplayName: Fijoy Vadakkumpadan
ThreadText: # How does DistilBERT's training ensure that the [CLS] token's hidden state from DistilBERT aligns with that from BERT?
  
  (Asked by: username_0 on 2024-12-08)
  
  BERT's pre-training involves next sentence prediction (NSP) based on a classifier on top of the [CLS] token's last hidden state. This is primarily what gives the [CLS] token's hidden state the special ability to represent the entire input, yes?
  
  DistilBERT's pre-training does not involve NSP. How can then the [CLS] token's hidden state from DistilBERT represent the entire input? I understand that DistilBERT's weights are initialized with corresponding weights from BERT, but is that enough?
  
  If the triple loss used in DistilBERT's distillation process aligns hidden states of *all* tokens with corresponding states from BERT, I can see how [CLS] token's hidden state would also be aligned. But [the original DistilBERT paper](https://arxiv.org/abs/1910.01108) seems to focus only on the masked tokens in the loss function, and the [CLS] token is not masked.
  
  **Tags:** bert, loss
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-08. Score: 1
  
  During pre-training, BERT learns to associate the [CLS] token’s hidden state with the semantic content of the entire input, and the classifier on top of the [CLS] token's representation helps BERT perform various downstream tasks like sentence classification.
  
  The distillation process focuses on aligning the hidden states of all tokens, including the [CLS] token via the required distillation loss which ensures that DistilBERT's representation of the [CLS] token is similar to the [CLS] token's representation in BERT. Also unlike the MLM loss, the additional optional cosine loss recommended by the original paper likely also focuses on aligning the hidden states of all tokens. Finally as you rightly mentioned, DistilBERT starts with BERT's weights initially, so the hidden state of the [CLS] token in DistilBERT is already in a good starting position to represent the entire input sequence in a similar manner to BERT. The distillation pretraining further refines and mimics this representation.
  
  
  ---
  
  