It is totally possible to store you forward passes and calculate the gradients later, as long as you keep the graph of the outputs (so no .detach()).
I think the error with the inplace operation has another origin. Do you have any ReLUs with inplace=True? Maybe turn them off.