That makes sense why I’ve always returned a gradient of 0, I’ve done some reading on Binarized Neural Networks 26. They overcome this issue through a quantized sign function and a hard tanh gradient. I’ve looked through various GitHub implementations of the suggested method; however, I can’t seem to figure out how to apply this to my threshold problem without returning a 0 gradient again. Is this a possible solution to my 0 gradient problem in autograd? Or should i try to approximate a probability using the hyperbolic tangent atan, which mostly returns a gradient larger than 0?