My solution has been to add this method:
def _custom_to(self,device):
    self.to(cg.device)
    for mod in self.modules():
        try:
            mod.__move_to_device__(device)
        except:
            pass

where the Module class has now a mod.__move_to_device__ that moves to the desired device the desired variables (in my case the non learnable parameter). I am not sure if there is a better way to do this. Also not sure if it worth working on incorporating this feature to the library. It would be as easy as adding the _custom_to() method and the __move_to__device. The later calls pass by default and can be overwritten by the user as I have done. The other solution is to call __move_to__device directly inside the standard nn.Module.to() method (which I think is better)