Resolved, pls check https://github.com/pytorch/pytorch/issues/21518 40, guess it’s still additive attention for numerical stability under the hood but behaves well(by replacing 0 as -inf before applying softmax for QK generated attention outputs).