Best solution for this would be for pytorch to provide similar interface to model.to(device) for the optimizer optim.to(device)  as well.
Another solution would have been to not save tensors in the state dicts with the device argument in them so that when loading a model would not result in this discrepancy between model state dict and optim state dict.
e.g. If all tensors in state dicts of both model and optim do not have a device assoctiated with them then simply doing
state_dict = torch.load('some_model.pt')
model.load_state_dict(statedict['model'])
optim.load_state_dict(state_dict['optim'])

would have worked fine .