Solved by ptrblck in post #9 
              
              
                A small update. I’m reading a tutorial on checkpointing models from the original author @Priya_Goyal  and this part seems to be interesting for your use case: 

NOTE: In case of checkpointing, if all the inputs don’t require grad but the outputs do, then if the inputs are passed as is, the output of…