Hi Chris @vdw,
I was wondering, how have your practices for padding changed since the rise of the transformer?
It seems to me that the best thing to do in that case it to pad with zero vector (equal to the size of the embed dim). That way with the self-attention, when we compare a real token and a padd token (which is just zero) the output attention results in being zero. Is this not what is done?