Do you mean plain Python variables by “CPU-stored (non-tensor) variables”, e.g. like x = torch.randn(1) * 1.0?
Generally you should transfer the data to the same device, if you are working with tensors.
However, you won’t see much difference, if you are using scalars, as the wrapping will be done automatically.