The -inf should not be multiplied with anything directly but only go into softmax. Typically attention dropout is applied afterwards (at least in bert/gpt).  Your application of dropout here is very nonstandard.
Best regards
Thomas