It seems there are different issues now.
If you are using ReLU as the activation function, it seems that you are somehow detaching (some) parameters, since conv1d_tdnn2.kernel.grad seems to be None suddently (while it works using tanh).
Anyway, using tanh returns that the gradients of all parameters are finite before the update step.
Nevertheless the next update creates invalid parameters.
The only reason I see now is that you might be dealing with an under/overflow of some values.
Could you check the max. absolute values of the gradients as well as parameters before the critical update step, which creates the NaNs?