Hi @albanD,
No, I do not know what to do then with respect to the input cut-off. And with my bias towards theory, a using discontinuous function seems unintuitive, too.
In fact, I prefer to think about this as shrinkage (i.e. relu(x-t), with its well-studied connections e.g. to a quadratic activation penalty or regression with noisy observation) plus a bias and don’t really like to think about thresholding. If you fed the output to (optionally) relu plus a layer that uses bias, I would think do not need the offset +t outside the relu at all.
But that’s me.
Best regards
Thomas