The variables returned by init_hidden are not registered with the module, so they are not moved to cpu when you call model.cpu(). PyTorch has this magic where if you assign a Parameter to a module object, it gets automatically registered. You need to turn the variables into parameters, and assign them separately onto self.