Could you print the shapes of proj_query and proj_key, as it seems that the bmm operation using these tensors is trying to create this huge tensor and I guess these tensors might be (accidentally) broadcasted.