If you are able to make your own autograd.Function, you can store things on CPU:
class StorethingsOnCPU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp):
        #...some calculation here...
        res = inp
        ctx.save_for_backward(inp.cpu())
        ctx._device = inp.device
        return res
    @staticmethod
    def backward(ctx, grad_out):
        inp_cpu, = ctx.saved_tensors
        inp = inp_cpu.to(ctx._device)
        #...do the backward here...
        print (inp_cpu, inp)
        grad_in = grad_out
        return grad_in

I don’t think there is a programmatic way to access the backwards of arbitrary built-in functions, though. At worst - and at the expense of a lot of time - you could re-do the forward on inp.detach().requires_grad_(), do a backward and return inp.grad.
Best regards
Thomas