You cannot use indexed assignment here, as the backward of the previous operation wants to use the calculation result you are overwriting.
The typical solution is to use torch.where(mask, attn_weights, neginf) where neginf is torch.tensor(float('-inf'), device=device) (I would recommend precomputing that).
I’m not sure your double-softmax makes much sense to me but that’s probably my limited imagination.
Best regards
Thomas