Wait, uh oh.  What I said is no longer true.  This worked for me with earlier versions of PyTorch, but now in PyTorch 0.4, this has stopped working.
It appears to work, but later when you’re training, you get an error from the optimizer’s (Adam in my case)  optimizer.step() method:
    exp_avg.mul_(beta1).add_(1 - beta1, grad)
RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #4 'other'

Can anyone describe how to load properly with 0.4, so this doesn’t happen?   Does one need to do optimizer.to(device) now, or something like that?  My code is the same, and everything is .cuda()'d before the model is saved, so I don’t see why it’s expecting a non-cuda Tensor.
UPDATE: Found an answer in this issue 528.
So after you load from the checkpoint, when you move your model to cuda, you need to move the optimizer values as well, like so:
   model, optimizer, start_epoch, losslogger = load_checkpoint(model, optimizer, losslogger)
    model = model.to(device)
    # now individually transfer the optimizer parts...
    for state in optimizer.state.values():
        for k, v in state.items():
            if isinstance(v, torch.Tensor):
                state[k] = v.to(device)

This works.  Is there a more elegant solution, @apaszke?