@PCerles @Felix_Kreuk
What @mrshenli mentioned could seamlessly happen when you load saved parameters without specifying map_location.
torch.load by default loads parameters to the device where they were, usually the rank 0 device.
load_state_dict then copies the loaded value from that device to the target device.
After the intermediate use, torch still occupies the GPU memory as cached memory.
I had a similar issue and solved it by directly loading parameters to the target device.
For example:
state_dict = torch.load(model_name, map_location=self.args.device)
self.load_state_dict(state_dict)

Full code here 55.