sigmoid uses the output to compute gradients:

  
      github.com
  
  
    pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml#L530-L531 21

- name: sigmoid(Tensor self)
self: _sigmoid_backward(grad, result)



  
  
    
    
  
  


relu uses the input to compute gradients:

  
      github.com
  
  
    pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml#L703-L704 10

- name: leaky_relu_forward(Tensor self, Scalar negative_slope)
self: leaky_relu_backward(grad, self, negative_slope)



  
  
    
    
  
  


Hence the different behaviors you see when modifying the output :). If you change



 serhii:

h[:, 0] = 0


in second example to
x[:, 0] = 0

, you will see a similar error.