Are you running out of memory or is it just a theoretical question?
What do you mean by reduce_grad for inputs?
You could have a look at torch.utils.checkpoint 84 to trade compute for memory or just lower your batch size, if that’s possible.