Hello Alex!



 googlebot:

sigmoid gates enforce convex/conic combinations
…
with tanh() it is basically not a gate anymore (but a source of oscillations)


Yes, I agree.  Your comment explains what is going on and the choice
of sigmiod().
In particular, with sigmoid():
z_t = torch.sigmoid(self.input_update(inp)+ self.hid_update(last_hid))

z_t is forced to be between 0 and 1, so that:
h_t = (1 - z_t) *n_t + z_t*last_hid

is, as you say, a convex combination of n_t and last_hid.
With tanh(), z_t can become negative, so that (assuming that h_t
becomes the new last_hid) last_hid can oscillate in sign.


as for relu, it is similarly non-gated rnn design.


With relu(), z_t can become larger than 1 so that last_hid can
grow exponentially (and also, (1 - z_t) can become negative).
Thanks for emphasizing the iterative structure of this use case.
Best.
K. Frank