I think you could create custom autograd.Functions as described here and store the tensors via ctx.save_for_backward and use them afterwards in the backward operation.
Since this new tensor would be stored the memory would depend on its size.