Add the GeLU activation from pytorch with the tanh approximation
Fixes #21344. See that issue for more details.
_The documentation is not available anymore as the PR was closed or merged._<|||||>Thanks for working on this! Does the new implementation in Pytorch produce the exact same results as `gelu_fast`? If that is the case, I would prefer we just replace the current `gelu_fast` with this when PyTorch is 1.12 or above.<|||||>> Thanks for working on this! Does the new implementation in Pytorch produce the exact same results as `gelu_fast`? If that is the case, I would prefer we just replace the current `gelu_fast` with this when PyTorch is 1.12 or above.

The results are similar but there are still rounding errors, see my analysis in the related issue #21344. I would also be in favor of replacing the existing implementation / using it as default, but I would introduce small numerical differences in some models, is that a problem?<|||||>Ah yes, the difference is quite significant sadly, so this will probably introduce a difference that is too big :-/
So let's go with a new activation. Maybe `gelu_pytorch` is a better name?<|||||>> Ah yes, the difference is quite significant sadly, so this will probably introduce a difference that is too big :-/ So let's go with a new activation. Maybe `gelu_pytorch` is a better name?

Wouldn't it cause confusion with the default pytorch implementation? That one is currently named "gelu". (And the one named "gelu_python").

Also should I add an explicit pytorch version check?
<|||||>Ok for the name then. For the version check, you will need to create a function that returns the instance of GELU and issues an import error if the  PyTorch version is too low, then put that function in the mappinh.<|||||>> Ok for the name then. For the version check, you will need to create a function that returns the instance of GELU and issues an import error if the PyTorch version is too low, then put that function in the mappinh.

Made a class to match the other activations, and raising a `NotImplementedError` (I don't think an `ImportError` is the best here since the function exists in earlier versions.) Also added to `test_get_activation`.<|||||>Failure is unrelated so merging. Thanks again for your contribution!