Hello,
This discussion sounds interesting to me because I was thinking the same.
Why there are different embedding vectors for PAD tokens.
My use-case is a multi-label text classification where I am using a pretrained model in MaskedLanguageModeling as an “embedding layer”. More specific, I feed the input text [b,t] padded to the “embedding layer” and it outputs [b,t,f], where b is the batch_size, t is the length of the max sequence in the batch, f is the feature_number.
After this I am using Attention to [b,t,f] and take a vector [b,1,f] which, after pass it from two linear layers and a sigmoid, gives the predictions.
I check cosine similarity between embedding vectors of PAD tokens and it is almost between all over 0.7. Additionally, cosine similarity between words’ embedding vectors and PAD tokens’ vectors is almost between all under 0.3.
Atenttion mechanism seems to assign negligible weights to PAD tokens embeddings vectors.
In general it seems that these vectors are kind of ignored from the model. Furthermore, my results are pretty ok with respect to accuracy.