Is it possible to keep intermediate results of forward for backward?
Hello, I would like to implement a new torch.autograd.Function where the gradient is closely related to an intermediate result. Is there a way to store the intermediate result for the backward pass to avoid having to compute it again? (Similar to save_for_backward, but that explicitly isn’t it…) Thank you Thomas
tom
smth
Thanks! <SCODE>RuntimeError: save_for_backward can only save input or output tensors, but argument 2 doesn't satisfy this condition
<ECODE>
tom
The autograd system is designed to handle intermediate results, but only if they were formulated using torch.autograd.Variable() and torch tensor functions, afaik. The error is because one of the arguments is not a Variable.
mxh000
The question is bit outdated, I encountered the same situation <SCODE>def forward(self, a, b, c):
     # okay
     # self.save_for_backward(a) 
              
     # wrong!!
     self.save_for_backward(a+1)
<ECODE> RuntimeError: save_for_backward can only save input or output tensors, but argument 2 doesn’t satisfy this condition
lenscloth
Best regards Thomas
tom
SivanK
I have the same question (I need to store info for backward). Unfortunately,  I don’t see where this requirement (store only input or output) comes from and how do I get around it (storing as a member in Function means you can use this function only once). Any ideas how to store intermediate things?
arogozhnikov
I see two main options. Best regards Thomas <SCODE>class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

<ECODE>
tom
Ok, probably it is a way to go, where do I find an example with instantiating function? Values that I need to store are actually integer numbers that occur during forward pass, so I can’t follow this recipe.
arogozhnikov
arogozhnikov