There’s nothing wrong with the ReLU implementation. It’s widely used and pretty simple:
(ReLU calls threshold)
CPU: https://github.com/pytorch/pytorch/blob/3b1c3996e1c82ca8f43af9efa196b33e36efee37/aten/src/ATen/native/cpu/Activation.cpp#L33
CUDA: https://github.com/pytorch/pytorch/blob/3b1c3996e1c82ca8f43af9efa196b33e36efee37/aten/src/ATen/native/cuda/Activation.cu#L290
The choice of derivative for torch.relu at 0 may vary between frameworks. The subgradient includes the interval [0,1]. PyTorch uses 0 for the derivative. I think TensorFlow also uses 0, but other frameworks might use 1. (Values between 0 and 1 are also subgradients, but would make for an awkward choice.)
                    
            
          


(Image from https://medium.com/@danqing/a-practical-guide-to-relu-b83ca804f1f7)