The answer for the why is in your own question.



 facehugger2020:

The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.


BERT implements a pooler output because it has been pretrained with an additional objective (next sentence prediction) that uses such output. RoBERTa does not use that objective anymore but it subclasses BERT in code so it also has that functionality. XLNet however is trained on PLM and is different altogether.
The BERT pooler is slightly different from what SequenceSummary does. You can compare the code if you want to. The pooler gets the output of CLS and puts it through a linear layer and TanH activation.

  
      github.com
  
  
    huggingface/transformers/blob/d5b3e56de5376aa85ef46e7f0325139d9e299a41/src/transformers/modeling_bert.py#L508-L520 2

class BertPooler(nn.Module):    def __init__(self, config):        super().__init__()        self.dense = nn.Linear(config.hidden_size, config.hidden_size)        self.activation = nn.Tanh()    def forward(self, hidden_states):        # We "pool" the model by simply taking the hidden state corresponding        # to the first token.        first_token_tensor = hidden_states[:, 0]        pooled_output = self.dense(first_token_tensor)        pooled_output = self.activation(pooled_output)        return pooled_output



  
  
    
    
  
  


The SequenceSummary has more options: you can choose the activation function, add dropout, and pick a summary type. This can explain the differences that you encounter.

  
      github.com
  
  
    huggingface/transformers/blob/f9a2a9e32bf4f6dc128decd0c124fa1f5507532e/src/transformers/modeling_utils.py#L1535-L1538 2

output = self.first_dropout(output)output = self.summary(output)output = self.activation(output)output = self.last_dropout(output)