@smth Is calling hard tanh in the forward() function of nn.Module equivalent to calling the derivative of hard tanh in the backward() function of nn.Function? This is for with respect to back propagation and automatic differentiation, so the backward pass. I believe there must be some difference, but I don’t really understand what it is.
Also, I notice that the inputs to this backward() function subclassed from nn.Function are insanely small. Is that supposed to be the case?
I am using the hard tanh as an activation function.