Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
one (think of text generation in ChatGPT), we need to apply the same rules during training. This is why we apply a causal mask to ensure that the targets, at each time step, can only see information from the past. Here is the corresponding section where the mask is applied (computing the mask is