Different seeds may significantly affect initial weights but not so much for fully trained model. Yes, you may stuck at some local minimum, but still, results should not be that different.
Since you are using transformers, have you tried experimenting with learning rate (value, warmups, annealing)?