Solved by Nathan_Wood in post #2 
              
              
                SGD does not keep track of extra variables relating to weights (unless you’re using momentum). This means you can simply create a new SGD optimizer. 
torch.save({'model': model.state_dict(), 'optim': optim.state_dict()}, '...')

To switch to SGD, use: 
state_dict = torch.load('...')
model.load_state…