Deep Learning, Sora, Computer Vision, Transformers, Attention.
tensor which we can pass through transformer layers and get the output. it is un-neccessary to calculate attention between two padded sequences. torch.nn.functional.scaled_dot_product_attention has attn_mask which will make sure that attn between masked portions is not calculated. we will see how