It seems that you can’t use numpy constants in such a construct, it leads it being stuck on the CPU :
def gelu(x):
	return 0.5 * x * (1 + F.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x*x*x)))

However if you replace the square root with 0.79788456080 everything works fine. Intentional?