Hi,
The problem is that self.action_std is not the Paramter, the .to() op creates a new Tensor.
You should do
        self.action_std = torch.nn.Parameter(torch.ones(action_dim, device=device)*action_std)