benoriol:

self.trainable_tensor = nn.Parameter(torch.rand(1, 2, 3)).to(device)


You have to move the tensor before creating the nn.Parameter.
i.e.,
self.trainable_tensor = nn.Parameter(torch.rand(1, 2, 3).to(device))

In your case, executing .to(device) after nn.Parameter creation has returned a tensor.
i.e., self.trainable_tensor was assigned with a tensor.
You can print the type of variable (type(self.trainable_tensor)) to make sure that you have created nn.Parameter correctly.