Hi, for a particular reason, I need to hand-derive a derivative for a GELU and it has to match the GELU implementation in Pytorch, but I tried finding the exact GELU formula and I’m a little confused. Could someone help simplify it for me by confirming whether the following is exactly equivalent to torch.nn.GELU:
0.5 * x * (1 + torch.tanh(np.math.sqrt(2 / np.math.pi) * (x + 0.044715 * torch.pow(x, 3)))
If not, could you provide the correct explicit formula? (Not for the derivative, just for the raw GELU)