Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
bit more crafting. There is something called Masked Multi-Head Attention. Remember what we said before about causal mask ? Well this happens here. We will use the attention_mask parameter of our Multi-head attention module to represent this (more details about how we compute the mask at the end) :