hi, sio277.
I use the following script based on pynvml to track memory usage.

Pytorch-Memory-Utils/gpu_mem_track.py at master · Oldpan/Pytorch-Memory-Utils · GitHub 2

I now want to use different loss values during the process. At the beginning of the training I use the following function to freeze the model, but when I unfreeze it during the training, it will report an error out of memory.
    @staticmethod
    def set_grads(mod, state):
        for para in mod.parameters():
            para.requires_grad = state

By the way, I set a flag (True or False) to select different loss values, but it will get error info (out of memory), and when the flag is canceled it will be fine.
I wonder why that is.
Looking forward to your reply :).