I am trying to make a Custom Relu function that just doesn’t apply the relu to the gradient.  The code is as follows:

class reluForward(torch.autograd.Function):
def forward(self, inp):
#option1:
#return = inp * (inp>0).float()
#option2:
#return F.relu(inp).data
#option3:
return F.relu(inp)
def backward(self, grad_out):
return grad_out

Option 1 and 2 both perform the correct function but are significantly slower than the built in nn.relu function (actually starting ok and then getting slower and slower as it runs) while option 3 tells me “data must be a Tensor.”  Looking for any ideas for the slowdown or a different way to do this!
Edit: I have also now tried with return inp.clamp(min=0) which also works with the slowdown.