May I know what do you mean by memory efficient?
As far as I know, Pytorch only stores the intermediate outputs for whatever weights, the gradient are needed (requires_grad=True). In one way, this seems to be doing what you ask for.
If I misunderstood your question incorrectly, can you explain with a small code, if possible?