Thanks, that makes sense.
I just did the test again, subtracting the max from X:
X = X - X.max(-1, keepdim=True)[0]
X = X.exp()
X = X / X.sum(-1, keepdim=True)

and I get the same time for the explicit calculation and the builtin version.