Hardtanh(0,0.5) has constant gradient, but it is almost the same as clamp; you have to ensure that pre-activation inputs in the inner region exist through the training - perhaps batchnorm could help here.