Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
attend only steps in the past, this means that for each time step T, the model can only attend to each step t for t in 1…T. It is a double for loop, we can therefore use a matrix to compute that : (image by author) def generate_square_subsequent_mask(size: int): """Generate a triangular (size,