Solved by SimonW in post #2 
              
              
                If this is not about the cached memory, use torch.autograd.grad instead of outputs.backward(xxxx, ...). See https://github.com/pytorch/pytorch/issues/7343 for details.