How do I compute/check/understand gradients of inplace ReLU? I did clone() to bypass “a leaf Variable that requires grad has been used in an in-place operation”. Gradient should obviously be 0, but I get 1.
import torch
import torch.nn.functional as F

# not in place
x = torch.tensor([-1.0]).requires_grad_().clone()
print(torch.autograd.grad(F.relu(x), (x, ))[0])
# tensor([0.])

# inplace
x = torch.tensor([-1.0]).requires_grad_().clone()
print(torch.autograd.grad(F.relu_(x), (x, ))[0])
# tensor([1.])

My final usecase is testing some ResNet-like code like following:
def residual_activation_inplace(x, residual, activation, inplace):
  for r in residual:
    x += r
  return activation(x, inplace = inplace)

x = torch.rand(3, 4)
residual = [torch.rand(3, 4), torch.rand(3, 4)]
y = residual_activation_inplace(x, residual, F.relu, inplace = True)