The Bert outputs two things :-
last_hidden_state: contains the hidden representations for each token in each sequence of the batch. So the size is (batch_size, seq_len, hidden_size).
pooler_output contains a “representation” of each sequence in the batch, and is of size (batch_size, hidden_size). What it basically does is take the hidden representation of the [CLS] token of each sequence in the batch
So suppose:-
hidden,pooled=model(...)
Shouldn’t
hidden[0] which is the embedding of 1st token ie: CLS be same as pooled as pooled is also the embedding of CLS?
While I have checked using code they both are not same. Why is it so? Am I missing something?
Thank you.