yeah and the difference is more drastic when you are not using bias. but even if you use bias, there are situations that the outputs are different… I guess there is no correct way to determine the derivative of ReLU at zero because you can define it either way.