If you freeze modules by setting the requires_grad attribute of their parameters to False, then you would save the memory the gradients would otherwise occupy (assuming the .grad attributes are set to None).