Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
# Stuff before self.self_attention = MultiHeadAttention(hidden_dim=n_dim, num_heads=n_heads) masked_att_output = self.self_attention( q=tgt, k=tgt, v=tgt, attention_mask=tgt_mask, <-- HERE IS THE CAUSAL MASK key_padding_mask=tgt_padding_mask) # Stuff after The second attention is called