Hmm, but how does Pytorch implement that integral? Is there no simple function equivalent to Pytorch’s implementation you could provide that I could enter into the following website:
http://www.matrixcalculus.org 1
I need to derive an MLP that uses GeLU activations and it would be convenient if I could just enter the full formula into that tool above.