Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
Stuff before self.cross_attention = MultiHeadAttention(hidden_dim=n_dim, num_heads=n_heads) cross_att_output = self.cross_attention( q=x1, k=memory, v=memory, attention_mask=None, <-- NO CAUSAL MASK HERE key_padding_mask=memory_padding_mask) <-- WE NEED TO USE THE PADDING OF THE SOURCE # Stuff