Solved by ptrblck in post #2 
              
              
                If you train all layers, it’s obviously expected, since you won’t save any computations. 
However, if you freeze some layers, you might save the backward computation, since you don’t need the gradients before a certain layer. 
In your use case, it seems you are retraining the input layer, thus the b…