The random numbers I get on the gpu are different than on the cpu:
In [17]: torch.manual_seed(123); torch.rand(device='cuda', size=(2,3))
Out[17]:
tensor([[ 0.0474,  0.1951,  0.2235],
        [ 0.5574,  0.7839,  0.9306]], device='cuda:0')

In [18]: torch.manual_seed(123); torch.rand(device='cuda', size=(2,3))
Out[18]:
tensor([[ 0.0474,  0.1951,  0.2235],
        [ 0.5574,  0.7839,  0.9306]], device='cuda:0')

In [19]: torch.manual_seed(123); torch.rand(device='cpu', size=(2,3))
Out[19]:
tensor([[ 0.2961,  0.5166,  0.2517],
        [ 0.6886,  0.0740,  0.8665]])

In [20]: torch.manual_seed(123); torch.rand(device='cpu', size=(2,3))
Out[20]:
tensor([[ 0.2961,  0.5166,  0.2517],
        [ 0.6886,  0.0740,  0.8665]])

One option could be to always generate on the cpu, then copy to the gpu. But presumably this characteristic is going to apply to eg dropout masks too? Another option would be to choose that seed consistency is only guaranteed conditioned on choice of using gpu or cpu.
What are best practices that are generally used to ensure reproducibility given a random seed?