During training you are using intermediate tensors needed to backpropagate and calculate the gradients. These intermediate tensors will be freed once the gradients were calculated (and you haven’t used retain_graph=True), so you’ll see more memory usage during training than the initial model parameters and buffers would use.