In DRLSTM you are using the device='cpu' argument as the default value and then apply it to self.bn, which would push this layer to the CPU. If you are passing a CUDATensor to this module, this error will be raised.
The better approach is not to specify the device as an argument and just calling model.to(device) after creating the object.
PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.