Solved by colesbury in post #2 
              
              
                There’s nothing wrong with the ReLU implementation. It’s widely used and pretty simple: 
(ReLU calls threshold) 
CPU: https://github.com/pytorch/pytorch/blob/3b1c3996e1c82ca8f43af9efa196b33e36efee37/aten/src/ATen/native/cpu/Activation.cpp#L33 
CUDA: https://github.com/pytorch/pytorch/blob/3b1c3996e1…