Hi Anuj!



 Anuj_Chopra:

In the function “gru_forward” there are 2 sigmoids and 1 tanh
…
if i replace the sigmoids with tanh at both places (all 3 tanh) then the network doesn’t learn (loss becomes nan).


This is more of a side comment than a direct answer:
Note that pytorch’s sigmoid() is the logistic function, and that is
a rescaled and shifted version of tanh().  Given that the weights
in Linear layers do scaling and their biases do shifts, you would
expect the two versions of your network to train to points where
sigmoid() and tanh() act essentially equivalently.
I would speculate that your network (together with its training data)
is close to being unstable, so that making the seemingly irrelevant
change from sigmoid() to tanh() is enough to kick it into an
unstable regime.
What happens if you use  a plain-vanilla SGD optimizer and/or lower
your learning rate?
relu() is different in character from sigmoid() / tanh(), but, even
so, I wouldn’t expect a mixture of relu() and tanh() to break your
training unless your network were already close to being unstable.
Best.
K. Frank