Solved by ptrblck in post #2 
              
              
                Your assumption is correct. During evaluation you don’t need to calculate the gradients, which would take memory, and also don’t need to store intermediate activations, which would be needed to calculate the gradients. 
To lower the memory usage and not store these intermediates, you should wrap you…