Are you using nn.DataParallel both for the CPU and GPU models? I had a similar issue and that fixed it for me.
Edit: Doesn’t look like you are. Try that adding nn.DataParallel(.) to wrap your model before loading the state dict.