Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
ValueError(f"Attention mask size {attention_mask.size()}") # Key mask here if key_padding_mask is not None: key_padding_mask = key_padding_mask.unsqueeze(1).unsqueeze(2) # Broadcast over batch size, num heads logits = logits + key_padding_mask attention = torch.softmax(logits, dim=-1) output =