Aloha Jonathan!



 VolcanoJ:

overwriting the Backwards function with 1 −  tanh^2(x)
However, I did not get the same results as when I used the autograd version of tanh’s derivative.


It works for me:
>>> import torch
>>> torch.__version__
'1.7.1'
>>> t = torch.arange (1.0, 4.0)
>>> t.requires_grad = True
>>> t_tanh = torch.tanh (t)
>>> t_tanh.sum().backward()
>>> t
tensor([1., 2., 3.], requires_grad=True)
>>> t_tanh
tensor([0.7616, 0.9640, 0.9951], grad_fn=<TanhBackward>)
>>> t.grad
tensor([0.4200, 0.0707, 0.0099])
>>> 1.0 - t_tanh**2
tensor([0.4200, 0.0707, 0.0099], grad_fn=<RsubBackward1>)
>>> t.grad - (1.0 - t_tanh**2)
tensor([0., 0., 0.], grad_fn=<SubBackward0>)

Could you tell us how you are invoking autograd?  (Also let us know
what version of pytorch you are using.)
Best.
K. Frank