This did not work for me. I am encountering the same issue. I am able to train on 1 fold, then when I reset state_dicts for model, optimizer and scheduler, I get OOM error at optimizer.step()
pytorch version '1.10.0+cu102'
If I understood correctly, you suggested
checkpoint = torch.load(..., map_location='cpu')
model.load_state_dict(checkpoint['state_dict'])
model = model.cuda()

This gets out of memory at optimizer.step() after training successfully on 1 fold.